Memory and Mixed Precision

Train a 1.3-billion-parameter model with AdamW in mixed precision. Its weights, gradients and optimizer state take 21 GB. One batch of 8 × 2,048 tokens adds 92 GB of activations. An 80 GB GPU says no.

Learn where every byte of training memory goes, how to read it on a real GPU, and the tools that shrink it: checkpointing, gradient accumulation, float16 and bfloat16, loss scaling, autocast and master weights.

Pick a model and a batch. The bar stacks the memory of one training step against the red line of your GPU. Then switch precision, optimizer, attention and checkpointing until it fits.

You need tensors and dtypes (part 1) and the training loop (part 5). We build everything else from zero.

The memory budget

Ready
1.3B
8
2,048
Precision
Optimizer
Attention
Checkpointing
GPU

GPT-style shapes: L layers of width h, head size 128, a 50,257-token vocabulary, 12·L·h² + 50,257·h parameters. Model states follow the ZeRO paper's accounting (16 bytes per parameter for mixed-precision Adam). Activations follow Korthikanti et al.'s per-layer formula sbh(34 + 5as/h) bytes for 16-bit activations, doubled for float32. Temporary: float32 logits and their gradient for the loss, plus 1 GB for the CUDA context and workspaces (illustrative). Fragmentation is not modelled.

Chapter 0

Step 1 runs. Step 2 crashes.

A run survives its first step and dies on the second with "CUDA out of memory". Follow the memory through one step to see why

You fine-tune a 1.3-billion-parameter language model on a 40 GB GPU. Batch 4, sequences of 2,048 tokens, AdamW, bfloat16 autocast, and a loop copied from an older tutorial that clears gradients with opt.zero_grad(set_to_none=False). Step 1 finishes. Step 2 stops with torch.OutOfMemoryError: CUDA out of memory. Nothing in your code changed between the two steps.

Something did change: the memory. GPU memory during training is not one number. It is a stack of separate piles, and each pile appears and disappears at its own moment in the step.

Step 1 has no optimizer state during its forward pass, because Adam creates its state lazily, inside the first step(). Step 2 carries that state through its whole forward and backward pass. Its peak is higher.

Mstates
The model states: parameters, gradients and optimizer state. The ZeRO paper uses this name.
Mact
The activations alive at the worst moment, which is the end of the forward pass.
Mtmp
Temporary buffers alive at that moment, for example the float32 logits of the loss.
Mpeak
The most memory the step ever holds at once. If it passes the GPU's size, the step fails.

Worked, step 2 of the run above: weights 5.2 GB + Adam state 10.5 GB + bfloat16 weight copies 2.6 GB + activations 13.7 GB + float32 logits and their gradient 3.3 GB = 35.3 GB. That fits in 40 GB, but only if the old gradients (another 5.2 GB) were freed. Keep them and the peak is 40.6 GB: out of memory.

Two steps, pile by pile

The area shows what the GPU holds through two training steps. Raise the batch until step 2 crosses the red line. Then switch zero_grad between freeing the gradients and zeroing them in place.

4
zero_grad
GPU

A 1.3B-parameter GPT-style model (24 layers, width 2,048) under bfloat16 autocast: float32 weights and gradients (4 bytes each per parameter), Adam's two float32 moments (8 bytes), autocast's cached bfloat16 weight copies (2 bytes) during forward and backward, flash-attention activations of 34·s·h bytes per layer per sequence (Korthikanti et al.), float32 logits for the loss. Each phase's shape is simplified; real curves are steppier.

Look at the moment of the peak. It is at the end of the forward pass, when every activation is alive and the backward pass has not freed any of them yet. The gradients of step 1 are still there too, unless zero_grad freed them.

That is the lesson's plan in miniature. First learn each pile: what it holds, its size per parameter or per token, and when it lives. Then learn the tools that shrink each pile, and the numerical traps that come with them.

The road map of this lesson

  1. Parameters, gradients, optimizer state: bytes per parameter, counted from a real state_dict. Build it
  2. The 16-byte rule: mixed precision's accounting, from the ZeRO paper. Why it works
  3. Activations: what one transformer layer saves, and why long sequences explode.
  4. Read memory on a real GPU: allocated, reserved, the peak, and leaks. Did it work?
  5. Activation checkpointing: trade compute for memory, √n segments. Build it
  6. Gradient accumulation: a big batch in small pieces, weighted right. Build it
  7. float16 and bfloat16: range against precision, and gradients that vanish. Why it works
  8. Loss scaling: GradScaler's rule, written by you. Build it
  9. Autocast: which ops run in low precision, and which stay float32. Did it work?
  10. Master weights, bfloat16 and 8-bit: updates that vanish, and smaller optimizers. When not
  11. Connections: a symptom guide, the cheat sheet and the rest of the series.

PyTorch, Zero to Expert

This is part 7 of an eleven-part series. Each lesson has Code Labs that run the PyTorch API in your browser.

  1. PyTorch Tensors: Storage, Strides and Views
  2. Autograd: the Graph Behind backward()
  3. nn.Module Inside Out
  4. Datasets and DataLoaders
  5. The Training Loop, Line by Line
  6. Debugging Training: When the Loss Won't Go Down
  7. Memory and Mixed Precision (this lesson)
  8. Making PyTorch Fast
  9. Distributed PyTorch: DDP and FSDP from the Inside
  10. PyTorch Internals: From a Python Call to a Kernel
  11. Build PyTorch from Scratch
Why can step 2 run out of memory when step 1 did not?

Chapter 1 Build it

Parameters, gradients, optimizer state

Count the bytes each parameter costs during training, for SGD, momentum and Adam, and check the count against a real model

Start with one parameter, one float32 number. It costs 4 bytes to store. Training adds more copies of the same size.

The backward pass writes a gradient into p.grad, with the same shape and dtype as p: 4 more bytes. Plain SGD needs nothing else: p -= lr * p.grad.

SGD with momentum keeps a running average of past gradients, the momentum buffer: 4 more bytes. Adam and AdamW keep two running averages: exp_avg, the mean of the gradients, and exp_avg_sq, the mean of their squares. That is 8 more bytes. PyTorch also stores a step counter per parameter tensor, a single float32 number, which is too small to matter.

N
The number of parameters: sum(p.numel() for p in model.parameters()).
bw
Bytes per weight: p.element_size(), 4 for float32.
bg
Bytes per gradient: the same as the weight's dtype, 4 for float32.
bopt
Optimizer bytes per parameter: 0 for SGD, 4 for momentum, 8 for Adam and AdamW.

Worked, 1.31 billion parameters with float32 AdamW: 1.31 × 109 × (4 + 4 + 8) = 21 GB. With plain SGD: 1.31 × 109 × 8 = 10.5 GB. The optimizer alone doubles the bill.

What one parameter costs

Pick an optimizer. Each box is one 4-byte float32 copy that training keeps per parameter. Then grow the model and see where it stops fitting.

Optimizer
1.3B

Model states only, in float32. Activations, temporary buffers and the CUDA context come on top.

Now check the rule on a real model instead of trusting it. Every pile is a set of tensors you can reach: model.parameters(), each p.grad, and opt.state, a dict from each parameter to its own state dict. A tensor's size in bytes is t.numel() * t.element_size().

One more kind of tensor lives in a model: a buffer. BatchNorm's running mean and variance are buffers. They are saved in state_dict() but get no gradient and no optimizer state. So state_dict() holds parameters plus buffers, and nothing about training.

A 7-billion-parameter model is trained in float32 with SGD and momentum. How much memory do the model states take?

Chapter 2 Why it works

The 16-byte rule

Mixed precision runs the maths in 16 bits but keeps float32 where precision matters. Derive its cost per parameter, and see what it does and does not save

A float32 number takes 4 bytes. A float16 or bfloat16 number takes 2. Modern GPUs multiply 16-bit matrices several times faster than float32 ones, and every 16-bit tensor moves half the bytes through memory. So why not train everything in 16 bits?

Because some numbers in training need more precision than 16 bits give. Micikevicius et al. (2017) found the fix that became standard: mixed precision. Run the forward and backward pass in 16 bits. Keep a float32 master copy of the weights, and let the optimizer update that copy. Chapter 10 shows why the master copy is needed.

Now count the bytes, the way the ZeRO paper does (Rajbhandari et al., 2019, Section 3.1). Call the number of parameters Ψ. The 16-bit weights take 2Ψ bytes. The 16-bit gradients take 2Ψ. The optimizer keeps three float32 tensors: the master weights, Adam's momentum and Adam's variance, 4Ψ each.

2Ψ
The 16-bit weights that the forward and backward pass read.
2Ψ
The 16-bit gradients that the backward pass writes.
K
The optimizer's bytes per parameter, which ZeRO calls the memory multiplier: float32 master weights, momentum and variance.
16Ψ
The model states of mixed-precision Adam, before any activation.

Worked, from the ZeRO paper: GPT-2 with 1.5 billion parameters needs 16 × 1.5 = 24 GB of model states, although its 16-bit weights are only 3 GB. A 7.5-billion-parameter model needs 120 GB, more than any single GPU of 2019.

Read that again: mixed-precision Adam costs 16 bytes per parameter, exactly what float32 Adam costs. Mixed precision does not shrink the model states. It moved bytes from the weights into the optimizer. What it does shrink is the activations, which are now 16-bit, and the time per step.

Five recipes, byte by byte

Pick a recipe. Each box is one copy kept per parameter; its width is its bytes, its colour its pile, its label its dtype. The bar below multiplies by the model size.

1.5B

fp16 mixed: the ZeRO paper's accounting (Section 3.1). torch autocast: PyTorch's native route keeps float32 parameters and gradients and casts weights to 16 bits inside each op; the cast copies are cached during the forward and backward pass. Pure bfloat16: everything 16-bit, no master copy (Chapter 10 shows the cost). 8-bit Adam: Dettmers et al. (2021) store each Adam state in 1 byte; the small per-block scale factors are left out.

PyTorch's own mixed precision, torch.autocast, reaches 16 bytes by a different road. The parameters stay float32: 4 bytes. Their gradients are float32 too: 4 bytes. Adam adds 8. Inside the autocast region each matrix multiply casts its float32 weight to bfloat16 or float16, and autocast keeps those casts for the rest of the region: about 2 more bytes per parameter while the forward and backward pass run.

You switch a 3B-parameter model from float32 AdamW to fp16 mixed precision with AdamW (master weights in float32). What happens to the model-state memory?

Chapter 3

Activations: what the forward pass keeps

Every op saves what its backward formula needs. Add up what one transformer layer saves, and see why batch, length and width multiply

The backward pass needs numbers from the forward pass. Take a linear layer y = x WT. Its weight gradient is gT x, where g is the gradient arriving from above. So the layer must keep its input x until the backward pass reaches it. Autograd does this for you: every op saves exactly what its backward formula needs (see Autograd).

These saved tensors are the activations. A linear layer saves its input. GELU saves its input. Softmax saves its output. Dropout saves its mask. LayerNorm saves its input and two small statistics. Each saved tensor has the shape of the data flowing through: batch × sequence × width.

Korthikanti et al. (2022, Section 4.1) added up one transformer layer, with 16-bit activations and 1-byte dropout masks. With b sequences of s tokens, width h and a attention heads:

s
The sequence length, in tokens.
b
The batch size, in sequences.
h
The hidden width of the model.
a
The number of attention heads. The term 5as/h comes from the a × s × s attention matrices: the softmax output, its dropout mask and the dropout output.
34
Everything else: inputs of the projections and the MLP (the MLP's 4h-wide hidden layer is most of it), the two LayerNorm inputs and the dropout masks, in units of s·b·h bytes.

Worked, the 1.3B model (h = 2,048, a = 16, 24 layers), b = 8, s = 2,048: s·b·h = 33.6 million; 34 + 5 × 16 × 2,048 / 2,048 = 34 + 80 = 114; per layer 33.6 M × 114 = 3.8 GB; × 24 layers = 92 GB. The attention matrices alone are 64 GB of it.

One layer's saved tensors

Each bar is one saved tensor of one layer of the 1.3B model. Double the sequence length and watch the three attention-matrix bars grow four times. Then switch on flash attention.

2,048
8

Item sizes from Korthikanti et al., Section 4.1 (16-bit activations, 1-byte dropout masks, small LayerNorm statistics ignored). Width 2,048, 16 heads, 24 layers. Models trained without dropout save a little less.

Three things follow from the formula. First, activations grow with b, s, h and the number of layers L, all multiplied together. Model states depend on none of b and s. So a batch that is too big fails on activations, never on weights.

Second, the attention matrices grow with s2. At s = 2,048 they are 70% of the layer; at s = 8,192 they are over 90%. Flash attention (Dao et al., 2022) computes attention in tiles and recomputes them in the backward pass, so the a × s × s matrices are never stored. PyTorch's F.scaled_dot_product_attention uses such a kernel on GPUs that support it. What is left is 34·s·b·h per layer: 27 GB in the worked example.

Third, the ZeRO paper's own estimate agrees. It counts about 12 × h × b × s × L 16-bit values for a GPT-2-like model: for the 1.5B GPT-2 (h = 1,600, 48 layers) at b = 32 and s = 1,024 that is 30 billion values, "about 60 GB" (Section 3.2).

With standard attention (the s × s matrices are stored), you double the sequence length from 2,048 to 4,096. What happens to the activation memory per layer of the 1.3B model?

Chapter 4 Did it work?

Read memory on a real GPU

Allocated, reserved, the peak and the snapshot: what each number means, why nvidia-smi disagrees, and how a leak looks

A budget on paper is a guess. Measure it. PyTorch counts every byte it hands out on the GPU, and four functions read those counts.

Why two numbers, allocated and reserved? Asking the CUDA driver for memory is slow, and giving it back forces the GPU to wait. So PyTorch uses a caching allocator. When a tensor dies, its block goes back to PyTorch's own cache, not to the driver. The next tensor of a similar size reuses it. Reserved is allocated plus the cache.

nvidia-smi shows the reserved memory plus the CUDA context, not what your tensors use. A process that looks full in nvidia-smi may have plenty of free blocks inside its cache. torch.cuda.empty_cache() returns the unused cached blocks to the driver; it frees no tensor.

before
Read right before the forward pass, after reset_peak_memory_stats(): the model states that live across steps.
peak
Read after the step: the highest point the step reached.
difference
What the step itself needed on top: activations, temporary buffers, gradients.

Worked, step 2 of Chapter 0 with set_to_none=True: before the forward pass 15.7 GB is allocated (weights 5.2 + Adam 10.5); the peak is 35.3 GB; so the step needs 19.6 GB on top. That is the number to compare with the free memory when you raise the batch.

Allocated, reserved, peak

Six training steps of a model. The solid area is memory_allocated, the outline memory_reserved, the dashed line max_memory_allocated. Turn on the leak, then try empty_cache.

An illustrative model: 6 GB of model states, 8 GB of activations per step, 1.5 GB of temporary buffers. The allocator keeps freed blocks; a third of the activation blocks are cut into sizes the next step cannot reuse whole (fragmentation). The leak keeps each step's graph alive: 3 GB per step here.

The leak that grows every step

Look at total_loss += loss. The loss is a tensor with a grad_fn. Adding it to a running total builds a graph that links every step's loss, and each loss holds on to its step's graph and saved activations. Memory grows every step until the run dies, often hours in. Write total_loss += loss.item(), or loss.detach(). The same leak hides in outputs.append(logits) for metrics.

Run this on a machine with PyTorch and a GPU

import torch

def step_memory(model, opt, batch, loss_fn):
    """Return (model states, extra peak for one step) in GB."""
    torch.cuda.synchronize()
    torch.cuda.reset_peak_memory_stats()
    before = torch.cuda.memory_allocated()       # model states (grads are None here)
    x, y = (t.cuda(non_blocking=True) for t in batch)
    with torch.autocast("cuda", dtype=torch.bfloat16):
        loss = loss_fn(model(x), y)
    loss.backward()
    opt.step()
    opt.zero_grad(set_to_none=True)
    torch.cuda.synchronize()
    peak = torch.cuda.max_memory_allocated()
    return before / 1e9, (peak - before) / 1e9

for i, batch in zip(range(3), loader):        # step 1 has no Adam state yet
    print(i, step_memory(model, opt, batch, loss_fn))
print(torch.cuda.memory_reserved() / 1e9, "GB reserved")

When the numbers do not explain an out-of-memory error, take a memory snapshot. It records every allocation with the Python stack that made it.

Run this on a machine with PyTorch and a GPU

torch.cuda.memory._record_memory_history(max_entries=100_000)
for _ in range(3):
    train_step()                                 # your step
torch.cuda.memory._dump_snapshot("snapshot.pickle")
torch.cuda.memory._record_memory_history(enabled=None)   # stop recording
# drag snapshot.pickle onto https://pytorch.org/memory_viz
nvidia-smi shows your process using 70 of 80 GB, and memory_allocated() says 30 GB. What is the most likely reading?

Chapter 5 Build it

Activation checkpointing

Keep a few activations, recompute the rest in the backward pass. Derive the √n rule, and prove the gradients do not change

Activations are kept only because the backward pass needs them. But anything the forward pass computed, it can compute again. Activation checkpointing (also called gradient checkpointing or recomputation) keeps only a few tensors, the checkpoints, and drops the rest. When the backward pass reaches a dropped region, it reruns that region's forward pass from the nearest checkpoint, then goes on.

How many checkpoints? Chen et al. (2016) worked it out. Split n layers into k segments of n/k layers. Keep only each segment's input. During the backward pass, rebuild one segment at a time. Count memory in units of one layer's activations:

n / k
The checkpoints: one stored input per segment. (Chen et al. write it O(k) with k segments; here k is the segment length, the same trade from the other side.)
k
The one segment rebuilt at a time during the backward pass: k layers' activations, alive together.
√n
Where the two terms are equal and their sum is smallest. Set the derivative −n/k2 + 1 to zero.

Worked, n = 64 layers: no checkpointing keeps 64 layers' activations. Segments of k = 8 keep 64/8 = 8 checkpoints and rebuild 8 layers at a time: 16, a quarter. The price is one extra forward pass. A backward pass costs about twice a forward pass, so the step goes from 3 units of work to 4: about 33% more. Chen et al. measured about 30% (Section 5.4), and cut a 1,000-layer ResNet from 48 GB to 7 GB.

Choose the segments

Each cell is one layer. Filled cells are checkpoints kept through the forward pass; the outlined segment is the one being rebuilt in the backward pass. Move the segment length and find the bottom of the curve.

64
4

Memory in units of one layer's saved activations; every layer the same size. k = 1 here means "keep every layer's input and rebuild one layer at a time", which saves nothing in this unit model but saves a lot in a transformer (below).

Transformers make checkpointing even better, because a layer's input is small next to what the layer saves inside. Checkpoint every layer: keep only each layer's input, 2·s·b·h bytes, instead of 34·s·b·h. Korthikanti et al. call this full recomputation (Section 5). For the 1.3B model at b = 8, s = 2,048 with flash attention: 24 inputs of 67 MB = 1.6 GB, plus one layer rebuilt (1.1 GB), instead of 27 GB.

import torch
from torch.utils.checkpoint import checkpoint

class Stack(torch.nn.Module):
    def __init__(self, layers, ckpt=True):
        super().__init__()
        self.layers = torch.nn.ModuleList(layers)
        self.ckpt = ckpt
    def forward(self, x):                        # x: (B, T, D)
        for layer in self.layers:
            if self.ckpt and self.training:
                x = checkpoint(layer, x, use_reentrant=False)   # keep only x
            else:
                x = layer(x)
        return x                                 # (B, T, D), same as without

Two settings matter. Pass use_reentrant=False: PyTorch 2.10 warns when you leave it out, and the old reentrant version silently trains nothing when no input of a segment needs a gradient. And leave preserve_rng_state=True, the default: it replays the same random numbers, so dropout draws the same mask the second time and the gradients stay exact.

Prove that last claim yourself. The lab counts which block outputs autograd still holds after the forward pass, using weak references that do not keep anything alive.

A 100-layer network, all layers the same size. With √n checkpointing, roughly how many layers' worth of activations are alive at the peak, and what does it cost?

Chapter 6 Build it

Gradient accumulation under a memory limit

Get the batch your recipe needs from micro-batches that fit, and weight every token the same when the micro-batches differ

Your recipe says batch 64. The GPU fits 4. Activation memory grows with the micro-batch, but the model states do not care about the batch at all. So run 16 small forward and backward passes, let the gradients add up in .grad, and take one optimizer step. That is gradient accumulation, and The Training Loop covers its basic form.

Here we use it as a memory tool. The micro-batch size m is set by memory; the number of micro-batches K is set by the batch you want: B = K · m. Peak memory follows m. The optimizer sees B.

Fit the batch

The 1.3B model at 2,048 tokens per sequence, mixed precision, flash attention. Pick the GPU and the batch the recipe needs. The bar is one micro-batch's memory; the readout says how many to accumulate.

GPU
64

Model states 21 GB plus 1 GB of context (Chapter 2). Per sequence: 3.4 GB of activations (0.34 GB when every layer is checkpointed) and 0.82 GB of float32 logits and their gradient. Micro-batch sizes are powers of two. The relative time assumes small micro-batches use the GPU less well (illustrative).

Accumulation is only exact if every example, or every token, carries the same weight it would carry in the big batch. With a mean loss and equal micro-batches, divide each loss by K. With token losses and padding, micro-batches hold different numbers of real tokens, and dividing by K is wrong.

ℓt
The loss of one real token t (padding positions are excluded with ignore_index=-100).
k
One micro-batch; the outer sum runs over all K of them.
Ntok
The number of real tokens in the whole batch. Count it before the loop.
L
The full-batch mean loss. Each micro-batch contributes loss_sum_k / N_tok to it.

Worked: two micro-batches with 100 and 300 real tokens. The fair weight of a token is 1/400. Per-micro-batch means divided by K = 2 give 1/(2 × 100) = 1/200 and 1/(2 × 300) = 1/600. A token in the short micro-batch counts 3 times as much as one in the long micro-batch.

Micro-batches of a token-level loss hold 50, 150 and 200 real tokens. What should each micro-batch's summed loss be divided by?

Chapter 7 Why it works

float16 and bfloat16: range against precision

Both formats use 16 bits, split differently. Find where gradients live on the number line, and which format loses them

A float has three parts: a sign bit, exponent bits that set how big or small a number can be (the range), and mantissa bits that set how finely numbers are spaced (the precision). PyTorch Tensors takes the bits apart. Here only the split matters.

FormatSign, exponent, mantissaLargestSmallest normalSmallest above 0Step near 1
float321, 8, 233.4 × 10381.2 × 10−381.4 × 10−451.2 × 10−7
float161, 5, 1065,5046.1 × 10−56.0 × 10−8 (2−24)9.8 × 10−4
bfloat161, 8, 73.4 × 10381.2 × 10−389.2 × 10−417.8 × 10−3

All values were checked with torch.finfo and by casting in PyTorch 2.10. float16 spends its bits on precision: steps 8 times finer than bfloat16. bfloat16 spends them on range: the same 8 exponent bits as float32, so anything float32 can hold, bfloat16 can hold roughly.

Now put gradients on that number line. Gradients are small. A loss averaged over many tokens divides every gradient by the token count. Micikevicius et al. plotted the activation gradients of an SSD object detector (Figure 3): much of float16's range was unused, while many values fell below it and became zero. That network did not train in float16 at all (Table 2: "diverges").

g
A gradient value, exact in float32.
fp16(g)
What float16 stores. Between 2−24 and 2−14 it keeps only a few bits (subnormal numbers); below half of 2−24 it stores 0: underflow.
S
A loss scale: multiply the loss by S and, by the chain rule, every gradient in the backward pass is multiplied by S too. A power of 2 shifts the exponent and changes no mantissa bit.

Worked: g = 1 × 10−8 is stored as 0. g = 4 × 10−8 is stored as 2−24 = 6.0 × 10−8: 49% wrong. With S = 212 = 4,096 both survive, and after dividing by S in float32 both are within 0.03%.

Where gradients live

The histogram is log2|g| of a layer's gradients. The bands are what each format can store. Slide the loss scale and watch the red (lost) part move into range, until the right tail overflows.

Format
0
-24

The histogram is an illustrative bell on a log scale, 7 binades wide (one standard deviation is 4 binades), loosely shaped like Micikevicius et al.'s Figure 3. Zero gradients are not drawn. Lost = below half the smallest step; coarse = subnormal, stored with fewer bits.

Two conclusions. First, float16 needs loss scaling for many models: it has the precision but not the range. bfloat16 has the range of float32, so gradients do not underflow, and it needs no scale (Chapter 10). Second, scaling has a ceiling: float16 stops at 65,504, so too big a scale turns the largest gradients into inf. The lab finds both edges.

Why do float16 gradients flush to zero while bfloat16 gradients of the same size do not?

Chapter 8 Build it

Loss scaling, and the scaler that tunes itself

A fixed scale works until it does not. Derive GradScaler's rule: back off on inf, grow after a calm stretch, never apply a poisoned step

The simplest fix is a constant loss scale. Micikevicius et al. trained networks with scales from 8 to 32K, and many needed none. If you know the largest gradient, pick S so that S times it stays under 65,504 (Section 3.2). Their warning: an overflow puts inf or NaN into the weight gradients, and one update with them damages the weights for good. So check the gradients, and skip the update when one is not finite.

But gradients change by orders of magnitude during training. A scale that is safe at step 100 wastes range at step 100,000. Dynamic loss scaling tunes S as it goes: try a big scale, back off when it overflows, and try bigger again after a stretch with no overflow. The paper names this idea as future work; PyTorch implements it as torch.amp.GradScaler.

S
The current scale. GradScaler starts at init_scale=65536.0, which is 216.
backoff
backoff_factor=0.5: halve on an overflow, and reset the count of good steps.
growth
growth_factor=2.0: double after a calm stretch.
growth_interval
growth_interval=2000 finite steps in a row before each doubling.

Worked, with the PyTorch 2.10 defaults: if the first three steps overflow, S falls 65,536 → 32,768 → 16,384 → 8,192 and three batches are skipped. If the run then stays calm, S returns to 16,384 after 2,000 more steps. Over 100,000 steps at most 50 doublings can happen, so a healthy run skips only a handful of steps.

Watch the scaler work

The largest gradient drifts down as training settles, with a few spikes. The teal line is log2 S; the purple line is the largest scaled gradient; red dots are skipped steps. Change the growth interval and the spikes.

100
few

GradScaler's exact update rule (backoff 0.5, growth 2.0) applied to an illustrative gradient history: the largest gradient falls from 2 to about 0.01 over 3,000 steps with random wobble, plus occasional spikes. A step overflows when the largest scaled gradient passes 65,504.

Here is the whole float16 training step. The order of the calls matters.

import torch
scaler = torch.amp.GradScaler("cuda")                # init_scale=2**16, growth_interval=2000

for x, y in loader:
    x, y = x.cuda(), y.cuda()
    with torch.autocast("cuda", dtype=torch.float16):
        loss = loss_fn(model(x), y)                  # forward + loss in mixed precision
    scaler.scale(loss).backward()                    # gradients are S times too big
    scaler.unscale_(opt)                             # divide by S in place, look for inf
    torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)   # clip TRUE gradients
    scaler.step(opt)                                 # skipped if an inf was found
    scaler.update()                                  # back off or grow S
    opt.zero_grad(set_to_none=True)

Now write the scaler yourself and race it against PyTorch's. The lab rounds every scaled gradient to float16 after the backward pass, so overflows happen for real.

GradScaler's scale is 32,768. Step 500 produces an inf in one gradient. What happens?

Chapter 9 Did it work?

Autocast: which ops go low, which stay float32

Autocast decides the dtype op by op. Learn its three lists, why softmax and the loss stay in float32 on a GPU, and how CPU differs

torch.autocast is a context manager. Inside it, PyTorch looks at each op as it runs and picks a dtype for it. You keep the model in float32 and change no layer. There are three lists.

Why must softmax stay in float32? Two reasons, range and sums. Softmax computes ex, and in float16 that overflows quickly:

ex
The exponential inside softmax, log_softmax, and every cross-entropy loss.
11.09
The largest x whose ex float16 can hold. Attention scores and logits pass this easily.

Worked: a logit of 12 gives e12 = 162,755: inf in float16, fine in float32 (whose limit is near e88.7). Softmax implementations subtract the row's maximum first, which helps, but the sum of a row still adds thousands of terms, and each 16-bit addition rounds. The loss averages over thousands of tokens the same way.

The autocast map

Each box is one op of a transformer block and its loss, coloured by the dtype it runs in. Switch between CUDA and CPU and watch the softmax. Notice which dtype the residual stream keeps.

Device
Low dtype

From PyTorch 2.10's autocast op reference (CUDA lists, CPU lists). The CPU results were also checked by running each op in PyTorch 2.10. The embedding lookup is on no list, so it returns the float32 table's dtype.

Notice the residual stream. The embedding table is float32, and every residual add mixes a float32 input with a 16-bit one, so promotion keeps the stream in float32 from the first layer to the last. Autocast never planned this; it falls out of the "follow your inputs" rule. Only the matrix multiplies inside each block drop to 16 bits.

The lab runs one attention step under CPU autocast. First predict each op's dtype, then fix the one place where CPU and CUDA differ that matters most: the softmax.

Inside torch.autocast("cuda", dtype=torch.float16), which ops run in float16?

Chapter 10 When not

Master weights, bfloat16 without a scaler, and 8-bit optimizers

Why a small update can vanish into a 16-bit weight, why bfloat16 skips loss scaling, and how to shrink the optimizer's bytes

Chapter 2 kept a float32 master copy of the weights and paid 4 bytes per parameter for it. Here is why. An update adds a small number Δ to a weight w. The sum must be rounded to the weight's format. If Δ is smaller than half the gap between w and its neighbour, the sum rounds back to w. The update is lost, every step, forever.

w
A weight stored in the low-precision format.
Δ
The update: −lr times the gradient for SGD, about −lr times ±1 for Adam.
ulp(w)
The unit in the last place: the gap between w and the next number the format can store. It grows with |w|.
m
Mantissa bits: 23 for float32, 10 for float16, 7 for bfloat16. Updates are lost once |w| / |Δ| passes about 2m+1: 2,048 for float16, 256 for bfloat16.

Worked: w = 1.0 and Δ = 0.001. In bfloat16 the gap at 1.0 is 2−7 = 0.0078; half of it, 0.0039, is bigger than 0.001, so 1.0 + 0.001 = 1.0. In float16 the gap is 0.00098, so the update survives (rounded to 0.00098). Micikevicius et al. give the float16 threshold as a ratio of 2,048 (Section 3.1); their Mandarin speech model lost 80% relative accuracy when it updated float16 weights directly.

The update that vanishes

The ticks are the numbers the format can store near w. The arrow is w + Δ. If the arrow ends closer to w than to the next tick, the update rounds away. Change the format, the weight and the update.

Weight format
1.0
0.001

Rounding is computed exactly: round to nearest, ties to even, as PyTorch does for float16 and bfloat16. Sliders are log2-spaced; the update is added to a positive weight.

In the lab, the weights start at 1.0 and must move by a few hundredths in small steps. Pure bfloat16 stalls; a float32 master copy keeps every small step.

bfloat16 needs no loss scaler

Loss scaling exists because float16's smallest number is 6 × 10−8. bfloat16 reaches 10−38 like float32, so gradients do not underflow, and its top is near 3 × 1038, so they do not overflow either. The recipe is just autocast:

with torch.autocast("cuda", dtype=torch.bfloat16):    # parameters stay float32: the master copy
    loss = loss_fn(model(x), y)
loss.backward()                                          # no GradScaler
opt.step()
opt.zero_grad(set_to_none=True)

Use float16 with a scaler only when the GPU has no fast bfloat16, as on NVIDIA GPUs older than the Ampere generation (for example V100 and T4). The other way round fails too: the PyTorch AMP docs warn that most bfloat16-pretrained models "cannot operate in the fp16 numerical range of max 65504" and overflow in float16.

Shrink the optimizer: 8-bit states

With the master copy settled, the biggest pile of model states is Adam's two moments: 8 bytes per parameter. Dettmers et al. (2021) store each moment in 8 bits. They split each state tensor into blocks of 2,048 values and quantize each block with its own scale, so one outlier does not ruin the precision of the whole tensor (Section 2.1). The states drop from 8 bytes to 2 per parameter: for a 1B-parameter model, from 8 GB to 2 GB. On their benchmarks, including a 1.5B-parameter language model, the 8-bit optimizers matched 32-bit Adam. The bitsandbytes library ships them as drop-in replacements such as bnb.optim.AdamW8bit.

Quantization goes further for inference and for fine-tuning: weights stored in 8 or 4 bits and expanded to 16 bits inside each matrix multiply. That is a lesson of its own: see Quantization I, sparsity and quantization, and Adapters and Fine-tuning Depth for training small adapters on top of a frozen, compressed model.

You train in bfloat16 with torch.autocast on a modern GPU. Do you need a GradScaler, and do you need a separate float32 master copy?

Chapter 11

Connections

Match a symptom to its cause, keep the one-screen cheat sheet, know the limits, and go on to speed and scale

Memory and precision bugs show up as a few symptoms. Pick one to see the usual causes in order, the check that tells them apart, and the fix.

From symptom to fix

Pick the symptom you see. The card lists the causes from most to least common, the one check to run first, and the fix.

A field guide from the chapters of this lesson, not an exhaustive list.

The cheat sheet

PartIn one line
PeakMstates + Mact + Mtmp; the peak is at the end of the forward pass; Adam's state exists from step 2.
Model statesfloat32: 4 + 4 + (0 SGD, 4 momentum, 8 Adam) bytes/param; mixed Adam: 2 + 2 + 12 = 16 (ZeRO); autocast: 4 + 4 + 8 (+2 cached casts).
Activationss·b·h(34 + 5as/h) bytes per transformer layer; flash attention drops the 5as/h; never depends on the optimizer.
Measurereset_peak_memory_stats(), memory_allocated() before, max_memory_allocated() after; reserved ≥ allocated; snapshot + memory_viz.
Checkpointingcheckpoint(fn, x, use_reentrant=False); n/k + k, best at k = √n; about +33% compute; identical gradients.
AccumulationB = K·m; m set by memory; token losses: reduction="sum" / Ntok of the whole batch.
float161-5-10 bits, max 65,504, smallest 2−24; needs loss scaling.
bfloat161-8-7 bits, float32's range, step 2−7 at 1; no scaler; keep float32 master weights.
GradScalerscale(loss).backward(), unscale_ before clipping, step skips on inf, update: ×0.5 on inf, ×2 after 2,000 good steps.
Autocastmatmul/conv low; on CUDA softmax, norms, sums, exp/log and losses float32; others follow inputs; CPU keeps softmax and norms in the input dtype.
Smaller states8-bit Adam: 2 bytes of state per parameter (blocks of 2,048); shard states across GPUs (ZeRO, FSDP).

Bugs experts have learned to spot

Honest limits

Where this sits, and where to go next

The takeaway. Training memory is model states plus activations plus temporaries. Model states cost about 16 bytes per parameter with Adam, in float32 or in mixed precision. Activations grow with batch, length, width and depth, and checkpointing, accumulation and flash attention shrink them. 16-bit maths needs float32 in three places: the master weights, the reductions, and, for float16, a loss scale that keeps gradients in range.
A run fails with out-of-memory in the first forward pass. The model states fit easily. What do you try first?

Now press Present or Teach and explain, out loud and from memory, where the 16 bytes per parameter go and why float16 needs a loss scale but bfloat16 does not. Then go back to the hero and make a 6.7B model train on one 80 GB GPU.

Memory and Mixed Precision, part 7 of PyTorch, Zero to Expert, in The Builder's Toolkit. Sources: S. Rajbhandari, J. Rasley, O. Ruwase, Y. He, "ZeRO: Memory Optimizations Toward Training Trillion Parameter Models" (2019, arXiv 1910.02054; Sections 3.1, 3.2, 5.1); T. Chen, B. Xu, C. Zhang, C. Guestrin, "Training Deep Nets with Sublinear Memory Cost" (2016, arXiv 1604.06174; abstract, Sections 4.3, 5.4); P. Micikevicius et al., "Mixed Precision Training" (ICLR 2018, arXiv 1710.03740; Sections 3.1 to 3.3, Figure 3, Table 2); V. Korthikanti et al., "Reducing Activation Recomputation in Large Transformer Models" (2022, arXiv 2205.05198; Sections 4.1, 5); T. Dettmers, M. Lewis, S. Shleifer, L. Zettlemoyer, "8-bit Optimizers via Block-wise Quantization" (ICLR 2022, arXiv 2110.02861; Sections 1.1, 2.1); T. Dao et al., "FlashAttention" (2022, arXiv 2205.14135). PyTorch 2.10 documentation: Automatic Mixed Precision (autocast op reference) and "CUDA semantics" (memory management). Every default, dtype, signature and limit quoted was checked by running PyTorch 2.10. Device numbers that are not from these sources are illustrative and labelled so.
Back to Gleams