Train a 1.3-billion-parameter model with AdamW in mixed precision. Its weights, gradients and optimizer state take 21 GB. One batch of 8 × 2,048 tokens adds 92 GB of activations. An 80 GB GPU says no.
Learn where every byte of training memory goes, how to read it on a real GPU, and the tools that shrink it: checkpointing, gradient accumulation, float16 and bfloat16, loss scaling, autocast and master weights.
Pick a model and a batch. The bar stacks the memory of one training step against the red line of your GPU. Then switch precision, optimizer, attention and checkpointing until it fits.
You need tensors and dtypes (part 1) and the training loop (part 5). We build everything else from zero.
GPT-style shapes: L layers of width h, head size 128, a 50,257-token vocabulary, 12·L·h² + 50,257·h parameters. Model states follow the ZeRO paper's accounting (16 bytes per parameter for mixed-precision Adam). Activations follow Korthikanti et al.'s per-layer formula sbh(34 + 5as/h) bytes for 16-bit activations, doubled for float32. Temporary: float32 logits and their gradient for the loss, plus 1 GB for the CUDA context and workspaces (illustrative). Fragmentation is not modelled.
Chapter 0
A run survives its first step and dies on the second with "CUDA out of memory". Follow the memory through one step to see why
You fine-tune a 1.3-billion-parameter language model on a 40 GB GPU. Batch 4, sequences of 2,048 tokens, AdamW, bfloat16 autocast, and a loop copied from an older tutorial that clears gradients with opt.zero_grad(set_to_none=False). Step 1 finishes. Step 2 stops with torch.OutOfMemoryError: CUDA out of memory. Nothing in your code changed between the two steps.
Something did change: the memory. GPU memory during training is not one number. It is a stack of separate piles, and each pile appears and disappears at its own moment in the step.
opt.step(). For Adam it is two extra numbers per parameter, and it stays for the rest of training.Step 1 has no optimizer state during its forward pass, because Adam creates its state lazily, inside the first step(). Step 2 carries that state through its whole forward and backward pass. Its peak is higher.
Worked, step 2 of the run above: weights 5.2 GB + Adam state 10.5 GB + bfloat16 weight copies 2.6 GB + activations 13.7 GB + float32 logits and their gradient 3.3 GB = 35.3 GB. That fits in 40 GB, but only if the old gradients (another 5.2 GB) were freed. Keep them and the peak is 40.6 GB: out of memory.
The area shows what the GPU holds through two training steps. Raise the batch until step 2 crosses the red line. Then switch zero_grad between freeing the gradients and zeroing them in place.
A 1.3B-parameter GPT-style model (24 layers, width 2,048) under bfloat16 autocast: float32 weights and gradients (4 bytes each per parameter), Adam's two float32 moments (8 bytes), autocast's cached bfloat16 weight copies (2 bytes) during forward and backward, flash-attention activations of 34·s·h bytes per layer per sequence (Korthikanti et al.), float32 logits for the loss. Each phase's shape is simplified; real curves are steppier.
Look at the moment of the peak. It is at the end of the forward pass, when every activation is alive and the backward pass has not freed any of them yet. The gradients of step 1 are still there too, unless zero_grad freed them.
That is the lesson's plan in miniature. First learn each pile: what it holds, its size per parameter or per token, and when it lives. Then learn the tools that shrink each pile, and the numerical traps that come with them.
This is part 7 of an eleven-part series. Each lesson has Code Labs that run the PyTorch API in your browser.
Chapter 1 Build it
Count the bytes each parameter costs during training, for SGD, momentum and Adam, and check the count against a real model
Start with one parameter, one float32 number. It costs 4 bytes to store. Training adds more copies of the same size.
The backward pass writes a gradient into p.grad, with the same shape and dtype as p: 4 more bytes. Plain SGD needs nothing else: p -= lr * p.grad.
SGD with momentum keeps a running average of past gradients, the momentum buffer: 4 more bytes. Adam and AdamW keep two running averages: exp_avg, the mean of the gradients, and exp_avg_sq, the mean of their squares. That is 8 more bytes. PyTorch also stores a step counter per parameter tensor, a single float32 number, which is too small to matter.
sum(p.numel() for p in model.parameters()).p.element_size(), 4 for float32.Worked, 1.31 billion parameters with float32 AdamW: 1.31 × 109 × (4 + 4 + 8) = 21 GB. With plain SGD: 1.31 × 109 × 8 = 10.5 GB. The optimizer alone doubles the bill.
Pick an optimizer. Each box is one 4-byte float32 copy that training keeps per parameter. Then grow the model and see where it stops fitting.
Model states only, in float32. Activations, temporary buffers and the CUDA context come on top.
Now check the rule on a real model instead of trusting it. Every pile is a set of tensors you can reach: model.parameters(), each p.grad, and opt.state, a dict from each parameter to its own state dict. A tensor's size in bytes is t.numel() * t.element_size().
One more kind of tensor lives in a model: a buffer. BatchNorm's running mean and variance are buffers. They are saved in state_dict() but get no gradient and no optimizer state. So state_dict() holds parameters plus buffers, and nothing about training.
Chapter 2 Why it works
Mixed precision runs the maths in 16 bits but keeps float32 where precision matters. Derive its cost per parameter, and see what it does and does not save
A float32 number takes 4 bytes. A float16 or bfloat16 number takes 2. Modern GPUs multiply 16-bit matrices several times faster than float32 ones, and every 16-bit tensor moves half the bytes through memory. So why not train everything in 16 bits?
Because some numbers in training need more precision than 16 bits give. Micikevicius et al. (2017) found the fix that became standard: mixed precision. Run the forward and backward pass in 16 bits. Keep a float32 master copy of the weights, and let the optimizer update that copy. Chapter 10 shows why the master copy is needed.
Now count the bytes, the way the ZeRO paper does (Rajbhandari et al., 2019, Section 3.1). Call the number of parameters Ψ. The 16-bit weights take 2Ψ bytes. The 16-bit gradients take 2Ψ. The optimizer keeps three float32 tensors: the master weights, Adam's momentum and Adam's variance, 4Ψ each.
Worked, from the ZeRO paper: GPT-2 with 1.5 billion parameters needs 16 × 1.5 = 24 GB of model states, although its 16-bit weights are only 3 GB. A 7.5-billion-parameter model needs 120 GB, more than any single GPU of 2019.
Read that again: mixed-precision Adam costs 16 bytes per parameter, exactly what float32 Adam costs. Mixed precision does not shrink the model states. It moved bytes from the weights into the optimizer. What it does shrink is the activations, which are now 16-bit, and the time per step.
Pick a recipe. Each box is one copy kept per parameter; its width is its bytes, its colour its pile, its label its dtype. The bar below multiplies by the model size.
fp16 mixed: the ZeRO paper's accounting (Section 3.1). torch autocast: PyTorch's native route keeps float32 parameters and gradients and casts weights to 16 bits inside each op; the cast copies are cached during the forward and backward pass. Pure bfloat16: everything 16-bit, no master copy (Chapter 10 shows the cost). 8-bit Adam: Dettmers et al. (2021) store each Adam state in 1 byte; the small per-block scale factors are left out.
PyTorch's own mixed precision, torch.autocast, reaches 16 bytes by a different road. The parameters stay float32: 4 bytes. Their gradients are float32 too: 4 bytes. Adam adds 8. Inside the autocast region each matrix multiply casts its float32 weight to bfloat16 or float16, and autocast keeps those casts for the rest of the region: about 2 more bytes per parameter while the forward and backward pass run.
Chapter 3
Every op saves what its backward formula needs. Add up what one transformer layer saves, and see why batch, length and width multiply
The backward pass needs numbers from the forward pass. Take a linear layer y = x WT. Its weight gradient is gT x, where g is the gradient arriving from above. So the layer must keep its input x until the backward pass reaches it. Autograd does this for you: every op saves exactly what its backward formula needs (see Autograd).
These saved tensors are the activations. A linear layer saves its input. GELU saves its input. Softmax saves its output. Dropout saves its mask. LayerNorm saves its input and two small statistics. Each saved tensor has the shape of the data flowing through: batch × sequence × width.
Korthikanti et al. (2022, Section 4.1) added up one transformer layer, with 16-bit activations and 1-byte dropout masks. With b sequences of s tokens, width h and a attention heads:
Worked, the 1.3B model (h = 2,048, a = 16, 24 layers), b = 8, s = 2,048: s·b·h = 33.6 million; 34 + 5 × 16 × 2,048 / 2,048 = 34 + 80 = 114; per layer 33.6 M × 114 = 3.8 GB; × 24 layers = 92 GB. The attention matrices alone are 64 GB of it.
Each bar is one saved tensor of one layer of the 1.3B model. Double the sequence length and watch the three attention-matrix bars grow four times. Then switch on flash attention.
Item sizes from Korthikanti et al., Section 4.1 (16-bit activations, 1-byte dropout masks, small LayerNorm statistics ignored). Width 2,048, 16 heads, 24 layers. Models trained without dropout save a little less.
Three things follow from the formula. First, activations grow with b, s, h and the number of layers L, all multiplied together. Model states depend on none of b and s. So a batch that is too big fails on activations, never on weights.
Second, the attention matrices grow with s2. At s = 2,048 they are 70% of the layer; at s = 8,192 they are over 90%. Flash attention (Dao et al., 2022) computes attention in tiles and recomputes them in the backward pass, so the a × s × s matrices are never stored. PyTorch's F.scaled_dot_product_attention uses such a kernel on GPUs that support it. What is left is 34·s·b·h per layer: 27 GB in the worked example.
Third, the ZeRO paper's own estimate agrees. It counts about 12 × h × b × s × L 16-bit values for a GPT-2-like model: for the 1.5B GPT-2 (h = 1,600, 48 layers) at b = 32 and s = 1,024 that is 30 billion values, "about 60 GB" (Section 3.2).
Chapter 4 Did it work?
Allocated, reserved, the peak and the snapshot: what each number means, why nvidia-smi disagrees, and how a leak looks
A budget on paper is a guess. Measure it. PyTorch counts every byte it hands out on the GPU, and four functions read those counts.
torch.cuda.memory_allocated(): the bytes held by live tensors right now.torch.cuda.max_memory_allocated(): the highest value of that since the program started, or since the last torch.cuda.reset_peak_memory_stats(). This is the number that decides out-of-memory.torch.cuda.memory_reserved(): the bytes PyTorch's allocator holds from the driver, used or not.torch.cuda.memory_summary(): a text table of all of these, by block size.Why two numbers, allocated and reserved? Asking the CUDA driver for memory is slow, and giving it back forces the GPU to wait. So PyTorch uses a caching allocator. When a tensor dies, its block goes back to PyTorch's own cache, not to the driver. The next tensor of a similar size reuses it. Reserved is allocated plus the cache.
nvidia-smi shows the reserved memory plus the CUDA context, not what your tensors use. A process that looks full in nvidia-smi may have plenty of free blocks inside its cache. torch.cuda.empty_cache() returns the unused cached blocks to the driver; it frees no tensor.
reset_peak_memory_stats(): the model states that live across steps.Worked, step 2 of Chapter 0 with set_to_none=True: before the forward pass 15.7 GB is allocated (weights 5.2 + Adam 10.5); the peak is 35.3 GB; so the step needs 19.6 GB on top. That is the number to compare with the free memory when you raise the batch.
Six training steps of a model. The solid area is memory_allocated, the outline memory_reserved, the dashed line max_memory_allocated. Turn on the leak, then try empty_cache.
An illustrative model: 6 GB of model states, 8 GB of activations per step, 1.5 GB of temporary buffers. The allocator keeps freed blocks; a third of the activation blocks are cut into sizes the next step cannot reuse whole (fragmentation). The leak keeps each step's graph alive: 3 GB per step here.
Look at total_loss += loss. The loss is a tensor with a grad_fn. Adding it to a running total builds a graph that links every step's loss, and each loss holds on to its step's graph and saved activations. Memory grows every step until the run dies, often hours in. Write total_loss += loss.item(), or loss.detach(). The same leak hides in outputs.append(logits) for metrics.
Run this on a machine with PyTorch and a GPU
import torch def step_memory(model, opt, batch, loss_fn): """Return (model states, extra peak for one step) in GB.""" torch.cuda.synchronize() torch.cuda.reset_peak_memory_stats() before = torch.cuda.memory_allocated() # model states (grads are None here) x, y = (t.cuda(non_blocking=True) for t in batch) with torch.autocast("cuda", dtype=torch.bfloat16): loss = loss_fn(model(x), y) loss.backward() opt.step() opt.zero_grad(set_to_none=True) torch.cuda.synchronize() peak = torch.cuda.max_memory_allocated() return before / 1e9, (peak - before) / 1e9 for i, batch in zip(range(3), loader): # step 1 has no Adam state yet print(i, step_memory(model, opt, batch, loss_fn)) print(torch.cuda.memory_reserved() / 1e9, "GB reserved")
When the numbers do not explain an out-of-memory error, take a memory snapshot. It records every allocation with the Python stack that made it.
Run this on a machine with PyTorch and a GPU
torch.cuda.memory._record_memory_history(max_entries=100_000) for _ in range(3): train_step() # your step torch.cuda.memory._dump_snapshot("snapshot.pickle") torch.cuda.memory._record_memory_history(enabled=None) # stop recording # drag snapshot.pickle onto https://pytorch.org/memory_viz
Chapter 5 Build it
Keep a few activations, recompute the rest in the backward pass. Derive the √n rule, and prove the gradients do not change
Activations are kept only because the backward pass needs them. But anything the forward pass computed, it can compute again. Activation checkpointing (also called gradient checkpointing or recomputation) keeps only a few tensors, the checkpoints, and drops the rest. When the backward pass reaches a dropped region, it reruns that region's forward pass from the nearest checkpoint, then goes on.
How many checkpoints? Chen et al. (2016) worked it out. Split n layers into k segments of n/k layers. Keep only each segment's input. During the backward pass, rebuild one segment at a time. Count memory in units of one layer's activations:
Worked, n = 64 layers: no checkpointing keeps 64 layers' activations. Segments of k = 8 keep 64/8 = 8 checkpoints and rebuild 8 layers at a time: 16, a quarter. The price is one extra forward pass. A backward pass costs about twice a forward pass, so the step goes from 3 units of work to 4: about 33% more. Chen et al. measured about 30% (Section 5.4), and cut a 1,000-layer ResNet from 48 GB to 7 GB.
Each cell is one layer. Filled cells are checkpoints kept through the forward pass; the outlined segment is the one being rebuilt in the backward pass. Move the segment length and find the bottom of the curve.
Memory in units of one layer's saved activations; every layer the same size. k = 1 here means "keep every layer's input and rebuild one layer at a time", which saves nothing in this unit model but saves a lot in a transformer (below).
Transformers make checkpointing even better, because a layer's input is small next to what the layer saves inside. Checkpoint every layer: keep only each layer's input, 2·s·b·h bytes, instead of 34·s·b·h. Korthikanti et al. call this full recomputation (Section 5). For the 1.3B model at b = 8, s = 2,048 with flash attention: 24 inputs of 67 MB = 1.6 GB, plus one layer rebuilt (1.1 GB), instead of 27 GB.
import torch from torch.utils.checkpoint import checkpoint class Stack(torch.nn.Module): def __init__(self, layers, ckpt=True): super().__init__() self.layers = torch.nn.ModuleList(layers) self.ckpt = ckpt def forward(self, x): # x: (B, T, D) for layer in self.layers: if self.ckpt and self.training: x = checkpoint(layer, x, use_reentrant=False) # keep only x else: x = layer(x) return x # (B, T, D), same as without
Two settings matter. Pass use_reentrant=False: PyTorch 2.10 warns when you leave it out, and the old reentrant version silently trains nothing when no input of a segment needs a gradient. And leave preserve_rng_state=True, the default: it replays the same random numbers, so dropout draws the same mask the second time and the gradients stay exact.
Prove that last claim yourself. The lab counts which block outputs autograd still holds after the forward pass, using weak references that do not keep anything alive.
Chapter 6 Build it
Get the batch your recipe needs from micro-batches that fit, and weight every token the same when the micro-batches differ
Your recipe says batch 64. The GPU fits 4. Activation memory grows with the micro-batch, but the model states do not care about the batch at all. So run 16 small forward and backward passes, let the gradients add up in .grad, and take one optimizer step. That is gradient accumulation, and The Training Loop covers its basic form.
Here we use it as a memory tool. The micro-batch size m is set by memory; the number of micro-batches K is set by the batch you want: B = K · m. Peak memory follows m. The optimizer sees B.
The 1.3B model at 2,048 tokens per sequence, mixed precision, flash attention. Pick the GPU and the batch the recipe needs. The bar is one micro-batch's memory; the readout says how many to accumulate.
Model states 21 GB plus 1 GB of context (Chapter 2). Per sequence: 3.4 GB of activations (0.34 GB when every layer is checkpointed) and 0.82 GB of float32 logits and their gradient. Micro-batch sizes are powers of two. The relative time assumes small micro-batches use the GPU less well (illustrative).
Accumulation is only exact if every example, or every token, carries the same weight it would carry in the big batch. With a mean loss and equal micro-batches, divide each loss by K. With token losses and padding, micro-batches hold different numbers of real tokens, and dividing by K is wrong.
ignore_index=-100).loss_sum_k / N_tok to it.Worked: two micro-batches with 100 and 300 real tokens. The fair weight of a token is 1/400. Per-micro-batch means divided by K = 2 give 1/(2 × 100) = 1/200 and 1/(2 × 300) = 1/600. A token in the short micro-batch counts 3 times as much as one in the long micro-batch.
Chapter 7 Why it works
Both formats use 16 bits, split differently. Find where gradients live on the number line, and which format loses them
A float has three parts: a sign bit, exponent bits that set how big or small a number can be (the range), and mantissa bits that set how finely numbers are spaced (the precision). PyTorch Tensors takes the bits apart. Here only the split matters.
| Format | Sign, exponent, mantissa | Largest | Smallest normal | Smallest above 0 | Step near 1 |
|---|---|---|---|---|---|
float32 | 1, 8, 23 | 3.4 × 1038 | 1.2 × 10−38 | 1.4 × 10−45 | 1.2 × 10−7 |
float16 | 1, 5, 10 | 65,504 | 6.1 × 10−5 | 6.0 × 10−8 (2−24) | 9.8 × 10−4 |
bfloat16 | 1, 8, 7 | 3.4 × 1038 | 1.2 × 10−38 | 9.2 × 10−41 | 7.8 × 10−3 |
All values were checked with torch.finfo and by casting in PyTorch 2.10. float16 spends its bits on precision: steps 8 times finer than bfloat16. bfloat16 spends them on range: the same 8 exponent bits as float32, so anything float32 can hold, bfloat16 can hold roughly.
Now put gradients on that number line. Gradients are small. A loss averaged over many tokens divides every gradient by the token count. Micikevicius et al. plotted the activation gradients of an SSD object detector (Figure 3): much of float16's range was unused, while many values fell below it and became zero. That network did not train in float16 at all (Table 2: "diverges").
Worked: g = 1 × 10−8 is stored as 0. g = 4 × 10−8 is stored as 2−24 = 6.0 × 10−8: 49% wrong. With S = 212 = 4,096 both survive, and after dividing by S in float32 both are within 0.03%.
The histogram is log2|g| of a layer's gradients. The bands are what each format can store. Slide the loss scale and watch the red (lost) part move into range, until the right tail overflows.
The histogram is an illustrative bell on a log scale, 7 binades wide (one standard deviation is 4 binades), loosely shaped like Micikevicius et al.'s Figure 3. Zero gradients are not drawn. Lost = below half the smallest step; coarse = subnormal, stored with fewer bits.
Two conclusions. First, float16 needs loss scaling for many models: it has the precision but not the range. bfloat16 has the range of float32, so gradients do not underflow, and it needs no scale (Chapter 10). Second, scaling has a ceiling: float16 stops at 65,504, so too big a scale turns the largest gradients into inf. The lab finds both edges.
Chapter 8 Build it
A fixed scale works until it does not. Derive GradScaler's rule: back off on inf, grow after a calm stretch, never apply a poisoned step
The simplest fix is a constant loss scale. Micikevicius et al. trained networks with scales from 8 to 32K, and many needed none. If you know the largest gradient, pick S so that S times it stays under 65,504 (Section 3.2). Their warning: an overflow puts inf or NaN into the weight gradients, and one update with them damages the weights for good. So check the gradients, and skip the update when one is not finite.
But gradients change by orders of magnitude during training. A scale that is safe at step 100 wastes range at step 100,000. Dynamic loss scaling tunes S as it goes: try a big scale, back off when it overflows, and try bigger again after a stretch with no overflow. The paper names this idea as future work; PyTorch implements it as torch.amp.GradScaler.
init_scale=65536.0, which is 216.backoff_factor=0.5: halve on an overflow, and reset the count of good steps.growth_factor=2.0: double after a calm stretch.growth_interval=2000 finite steps in a row before each doubling.Worked, with the PyTorch 2.10 defaults: if the first three steps overflow, S falls 65,536 → 32,768 → 16,384 → 8,192 and three batches are skipped. If the run then stays calm, S returns to 16,384 after 2,000 more steps. Over 100,000 steps at most 50 doublings can happen, so a healthy run skips only a handful of steps.
The largest gradient drifts down as training settles, with a few spikes. The teal line is log2 S; the purple line is the largest scaled gradient; red dots are skipped steps. Change the growth interval and the spikes.
GradScaler's exact update rule (backoff 0.5, growth 2.0) applied to an illustrative gradient history: the largest gradient falls from 2 to about 0.01 over 3,000 steps with random wobble, plus occasional spikes. A step overflows when the largest scaled gradient passes 65,504.
Here is the whole float16 training step. The order of the calls matters.
import torch scaler = torch.amp.GradScaler("cuda") # init_scale=2**16, growth_interval=2000 for x, y in loader: x, y = x.cuda(), y.cuda() with torch.autocast("cuda", dtype=torch.float16): loss = loss_fn(model(x), y) # forward + loss in mixed precision scaler.scale(loss).backward() # gradients are S times too big scaler.unscale_(opt) # divide by S in place, look for inf torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0) # clip TRUE gradients scaler.step(opt) # skipped if an inf was found scaler.update() # back off or grow S opt.zero_grad(set_to_none=True)
Now write the scaler yourself and race it against PyTorch's. The lab rounds every scaled gradient to float16 after the backward pass, so overflows happen for real.
Chapter 9 Did it work?
Autocast decides the dtype op by op. Learn its three lists, why softmax and the loss stay in float32 on a GPU, and how CPU differs
torch.autocast is a context manager. Inside it, PyTorch looks at each op as it runs and picks a dtype for it. You keep the model in float32 and change no layer. There are three lists.
linear, matmul, bmm, conv2d, scaled_dot_product_attention, …). These are most of the arithmetic and gain the most from 16-bit tensor cores. They are safe because the hardware adds the products up in float32 (Micikevicius et al., Section 3.3).softmax, log_softmax, layer_norm, group_norm, sum, exp, log, pow, and the losses (cross_entropy, mse_loss, …).Why must softmax stay in float32? Two reasons, range and sums. Softmax computes ex, and in float16 that overflows quickly:
Worked: a logit of 12 gives e12 = 162,755: inf in float16, fine in float32 (whose limit is near e88.7). Softmax implementations subtract the row's maximum first, which helps, but the sum of a row still adds thousands of terms, and each 16-bit addition rounds. The loss averages over thousands of tokens the same way.
Each box is one op of a transformer block and its loss, coloured by the dtype it runs in. Switch between CUDA and CPU and watch the softmax. Notice which dtype the residual stream keeps.
From PyTorch 2.10's autocast op reference (CUDA lists, CPU lists). The CPU results were also checked by running each op in PyTorch 2.10. The embedding lookup is on no list, so it returns the float32 table's dtype.
Notice the residual stream. The embedding table is float32, and every residual add mixes a float32 input with a 16-bit one, so promotion keeps the stream in float32 from the first layer to the last. Autocast never planned this; it falls out of the "follow your inputs" rule. Only the matrix multiplies inside each block drop to 16 bits.
The lab runs one attention step under CPU autocast. First predict each op's dtype, then fix the one place where CPU and CUDA differ that matters most: the softmax.
torch.autocast("cuda", dtype=torch.float16), which ops run in float16?Chapter 10 When not
Why a small update can vanish into a 16-bit weight, why bfloat16 skips loss scaling, and how to shrink the optimizer's bytes
Chapter 2 kept a float32 master copy of the weights and paid 4 bytes per parameter for it. Here is why. An update adds a small number Δ to a weight w. The sum must be rounded to the weight's format. If Δ is smaller than half the gap between w and its neighbour, the sum rounds back to w. The update is lost, every step, forever.
Worked: w = 1.0 and Δ = 0.001. In bfloat16 the gap at 1.0 is 2−7 = 0.0078; half of it, 0.0039, is bigger than 0.001, so 1.0 + 0.001 = 1.0. In float16 the gap is 0.00098, so the update survives (rounded to 0.00098). Micikevicius et al. give the float16 threshold as a ratio of 2,048 (Section 3.1); their Mandarin speech model lost 80% relative accuracy when it updated float16 weights directly.
The ticks are the numbers the format can store near w. The arrow is w + Δ. If the arrow ends closer to w than to the next tick, the update rounds away. Change the format, the weight and the update.
Rounding is computed exactly: round to nearest, ties to even, as PyTorch does for float16 and bfloat16. Sliders are log2-spaced; the update is added to a positive weight.
In the lab, the weights start at 1.0 and must move by a few hundredths in small steps. Pure bfloat16 stalls; a float32 master copy keeps every small step.
Loss scaling exists because float16's smallest number is 6 × 10−8. bfloat16 reaches 10−38 like float32, so gradients do not underflow, and its top is near 3 × 1038, so they do not overflow either. The recipe is just autocast:
with torch.autocast("cuda", dtype=torch.bfloat16): # parameters stay float32: the master copy loss = loss_fn(model(x), y) loss.backward() # no GradScaler opt.step() opt.zero_grad(set_to_none=True)
Use float16 with a scaler only when the GPU has no fast bfloat16, as on NVIDIA GPUs older than the Ampere generation (for example V100 and T4). The other way round fails too: the PyTorch AMP docs warn that most bfloat16-pretrained models "cannot operate in the fp16 numerical range of max 65504" and overflow in float16.
With the master copy settled, the biggest pile of model states is Adam's two moments: 8 bytes per parameter. Dettmers et al. (2021) store each moment in 8 bits. They split each state tensor into blocks of 2,048 values and quantize each block with its own scale, so one outlier does not ruin the precision of the whole tensor (Section 2.1). The states drop from 8 bytes to 2 per parameter: for a 1B-parameter model, from 8 GB to 2 GB. On their benchmarks, including a 1.5B-parameter language model, the 8-bit optimizers matched 32-bit Adam. The bitsandbytes library ships them as drop-in replacements such as bnb.optim.AdamW8bit.
Quantization goes further for inference and for fine-tuning: weights stored in 8 or 4 bits and expanded to 16 bits inside each matrix multiply. That is a lesson of its own: see Quantization I, sparsity and quantization, and Adapters and Fine-tuning Depth for training small adapters on top of a frozen, compressed model.
Chapter 11
Match a symptom to its cause, keep the one-screen cheat sheet, know the limits, and go on to speed and scale
Memory and precision bugs show up as a few symptoms. Pick one to see the usual causes in order, the check that tells them apart, and the fix.
Pick the symptom you see. The card lists the causes from most to least common, the one check to run first, and the fix.
A field guide from the chapters of this lesson, not an exhaustive list.
| Part | In one line |
|---|---|
| Peak | Mstates + Mact + Mtmp; the peak is at the end of the forward pass; Adam's state exists from step 2. |
| Model states | float32: 4 + 4 + (0 SGD, 4 momentum, 8 Adam) bytes/param; mixed Adam: 2 + 2 + 12 = 16 (ZeRO); autocast: 4 + 4 + 8 (+2 cached casts). |
| Activations | s·b·h(34 + 5as/h) bytes per transformer layer; flash attention drops the 5as/h; never depends on the optimizer. |
| Measure | reset_peak_memory_stats(), memory_allocated() before, max_memory_allocated() after; reserved ≥ allocated; snapshot + memory_viz. |
| Checkpointing | checkpoint(fn, x, use_reentrant=False); n/k + k, best at k = √n; about +33% compute; identical gradients. |
| Accumulation | B = K·m; m set by memory; token losses: reduction="sum" / Ntok of the whole batch. |
| float16 | 1-5-10 bits, max 65,504, smallest 2−24; needs loss scaling. |
| bfloat16 | 1-8-7 bits, float32's range, step 2−7 at 1; no scaler; keep float32 master weights. |
| GradScaler | scale(loss).backward(), unscale_ before clipping, step skips on inf, update: ×0.5 on inf, ×2 after 2,000 good steps. |
| Autocast | matmul/conv low; on CUDA softmax, norms, sums, exp/log and losses float32; others follow inputs; CPU keeps softmax and norms in the input dtype. |
| Smaller states | 8-bit Adam: 2 bytes of state per parameter (blocks of 2,048); shard states across GPUs (ZeRO, FSDP). |
total_loss += loss or outputs.append(logits): every step's graph stays alive.scaler.unscale_(opt): the threshold is compared with S-times-too-big gradients.model.half() plus autocast plus a GradScaler: float16 gradients and no master weights; GradScaler refuses with "Attempting to unscale FP16 gradients."use_reentrant=True and no input that needs a gradient: the parameters inside never learn.nvidia-smi as tensor memory: it shows the allocator's reserve plus the CUDA context.torch.cuda.synchronize() and reset_peak_memory_stats(), or only at step 1, before Adam's state exists.Now press Present or Teach and explain, out loud and from memory, where the 16 bytes per parameter go and why float16 needs a loss scale but bfloat16 does not. Then go back to the hero and make a 6.7B model train on one 80 GB GPU.