nn.Module Inside Out

Every PyTorch model is a tree of modules. Each module keeps three lists: its parameters, its buffers and its child modules. Training, saving, loading, freezing and moving to a GPU all walk that tree. Learn the tree, and most model bugs become easy to see.

Learn how a module records what you assign to it, what goes into a checkpoint, what train() and eval() change, how to freeze part of a model, how to look inside with hooks, and how to run a model with weights it does not own. Eight Code Labs run real PyTorch code in your browser.

Click any row of the tree. Freeze a branch, or switch the model to eval mode. Watch what trains, what is saved, and what changes.

You need tensors and autograd: see Tensors and Autograd. We build everything else from zero.

Open the model

Ready

TinyNet is a real model: two convolution blocks with BatchNorm, then a dropout and linear head for 10 classes. Every name, shape and count here is what PyTorch 2.10 prints for it. Click a row to select it; click its arrow to fold it.

Chapter 0 Bug to spot

The bug that trains fine

A model with a hidden mistake trains, the loss falls, and the saved checkpoint is still broken. Find out why, then see the map of this lesson

You write a small network. It has an input layer, three hidden layers and an output layer. You keep the hidden layers in a Python list, because you want to loop over them. You train it. The loss falls. You save the weights and go home.

The next day you load the weights into a fresh copy of the model. Its predictions are nonsense. Nothing raised an error. The loss curve looked healthy. What went wrong?

PyTorch never saw the hidden layers. A plain Python list is invisible to the model. So the optimizer did not train those layers, and the checkpoint did not save them. The fresh copy got new random hidden layers. Try the three ways to store the layers below.

Where did my layers go?

Pick how the hidden layers are stored. The network trains live in your browser. The left side shows what the model registered. The right side shows the training loss, then the loss of a copy reloaded from the checkpoint.

A 2 → 16 → 16 → 16 → 1 tanh network fits y = sin(3x0) cos(2x1) on 256 points with Adam (lr 0.01, 300 steps), trained in JavaScript with the same maths as PyTorch. "Reload" builds a fresh model and copies in only what the state_dict holds. Loss values are illustrative.

The loss curve is the trap. Gradients still flow through the hidden layers, so the input and output layers learn to work around them. Training looks fine. Only the reload shows the damage. This is why experts check what a model registered, not just whether the loss falls.

The road map of this lesson

Each chapter takes one job of nn.Module, shows how it works inside, and names the bugs experts look for.

  1. How a module sees what you assign: __setattr__ and the three registries. Lab: write a layer from scratch.
  2. Parameter, buffer or plain tensor: what each one survives. Lab: find the Python-list bug.
  3. Walk the tree: children, modules, named_parameters and the dotted names.
  4. Save and load: state_dict, strict loading, missing and unexpected keys. Lab: load a renamed checkpoint.
  5. train() and eval(): what Dropout and BatchNorm change. Lab: BatchNorm by hand.
  6. Freeze and split the learning rate: requires_grad_ and parameter groups. Lab: freeze a backbone, give the head 10× the LR.
  7. Initialisation: the defaults, why they shrink a deep ReLU stack, and apply. Lab: re-initialise a network.
  8. Hooks: read or change any layer's input and output. Lab: a feature extractor.
  9. Your own layer, and the containers: extra_repr, Sequential, ModuleList, ModuleDict.
  10. Stateless calls, and what a model costs: functional_call, ensembles, meta-learning and memory. Lab: an ensemble from one module.
  11. Connections: a symptom-to-cause card, the cheat sheet, and the rest of the series.

PyTorch, Zero to Expert

This is part 3 of an eleven-part series in The Builder's Toolkit.

  1. Tensors: Storage, Strides and Views
  2. Autograd: the Graph Behind backward()
  3. nn.Module Inside Out (this lesson)
  4. Datasets and DataLoaders
  5. The Training Loop, Line by Line
  6. Debugging Training: When the Loss Won't Go Down
  7. Memory and Mixed Precision
  8. Making PyTorch Fast
  9. Distributed PyTorch: DDP and FSDP from the Inside
  10. PyTorch Internals: From a Python Call to a Kernel
  11. Build PyTorch from Scratch
The hidden layers sit in a plain Python list. Training loss falls. What is still broken?

Chapter 1 How it works

How a module sees what you assign

Every self.x = … inside a module goes through one method that sorts the value into a registry. Watch it sort, then write a layer from scratch

Think of a module as a desk with labelled trays. When you write self.fc = nn.Linear(4, 2), you do not just set an attribute. You hand the value to a clerk. The clerk looks at it and drops it into the right tray. Everything PyTorch does later reads the trays, not your attributes.

The clerk is nn.Module.__setattr__. Python calls it for every assignment of the form self.name = value. It checks the value's type and sorts it:

That is why super().__init__() must come first in your __init__. It creates the empty trays. Assign a layer before it, and PyTorch raises AttributeError: cannot assign module before Module.__init__() call.

The sorting desk

Pick a line of __init__. The lit path shows the checks __setattr__ makes. The value lands in a tray. Only the first three trays are seen by parameters(), state_dict() and .to().

The checks follow torch/nn/modules/module.py. register_buffer is a method call, not an assignment, so it writes the buffer tray directly. Once a name is a buffer, a later self.mean = tensor goes through __setattr__ and updates that buffer.

Count what a layer owns

A linear layer computes y = x WT + b. Its weight W has one row per output and one column per input. Its bias b has one number per output. So its size is easy to count:

nin
Input features: the length of each input vector.
nout
Output features: the number of rows of W, and the length of b.
P
The number of trainable numbers in the layer: weight.numel() + bias.numel().

Worked, a 784 → 256 → 10 network: 784 × 256 + 256 = 200,960; 256 × 10 + 10 = 2,570; total 203,530. The ReLU between them owns nothing.

Here is a linear layer written from scratch. Read it line by line; then build it in the lab.

import math, torch, torch.nn as nn

class MyLinear(nn.Module):
    def __init__(self, n_in, n_out):
        super().__init__()                         # creates the empty registries
        self.n_in, self.n_out = n_in, n_out        # plain ints: kept, not registered
        bound = 1 / math.sqrt(n_in)                # PyTorch's default range
        self.weight = nn.Parameter(torch.empty(n_out, n_in).uniform_(-bound, bound))
        self.bias = nn.Parameter(torch.empty(n_out).uniform_(-bound, bound))

    def forward(self, x):                      # x: (B, n_in) -> (B, n_out)
        return x @ self.weight.T + self.bias

You never call forward yourself. You call the module: layer(x). That goes through Module.__call__, which runs any hooks (Chapter 8) and then forward. Calling layer.forward(x) directly skips the hooks.

Inside __init__ you write self.alpha = torch.ones(3, requires_grad=True). What happens?

Chapter 2 How it works Bug to spot

Parameter, buffer or plain tensor

Five everyday operations treat each kind of attribute differently. Learn the grid once, then fix the Python-list bug for real

A module holds three kinds of tensor. They look alike when you print them. They behave very differently when you train, save or move the model.

A buffer can also be non-persistent: register_buffer(name, t, persistent=False). It moves with the model but is left out of the checkpoint. Use it for things you can rebuild, such as a cached table of sines for positional encodings.

What each kind survives

Pick an operation. Each row is one way to hold a tensor or a layer. A filled mark means the operation reaches it.

Each mark was checked by running the operation in PyTorch 2.10. "backward()" means the tensor gets a .grad. "opt.step()" assumes the optimizer was built from model.parameters().

Look at the "layer in a Python list" row under backward(). Its weights do get gradients, because they are real Parameters used in the forward pass. But the optimizer was built from model.parameters(), which never yields them. So the gradients pile up and nothing ever uses them. That is the bug from Chapter 0. .double() also skips them, so the model ends up with mixed dtypes and crashes in the forward pass.

The fix is one word: nn.ModuleList. It is a list that is also a module, so __setattr__ files it under _modules, and its items get names blocks.0, blocks.1 and so on.

You need a lookup table that moves to the GPU with the model, but you rebuild it at start-up and do not want it in checkpoints. What do you use?

Chapter 3 How it works

Walk the tree: names and counts

Every walk over a model visits the same tree in the same order. Learn the walks, the dotted names they produce, and how to count what a model owns

Since a model is a tree, most of the nn.Module API is a walk over it. The walk is always the same: visit a module, then each child in the order you assigned them, depth first. What changes is what each walk yields.

CallYieldsFor TinyNet
children()Direct child modules only.2: backbone, head
modules()Every module, the root first.13
named_parameters()(dotted name, Parameter) pairs.10 tensors, 5,514 numbers
named_buffers()(dotted name, buffer) pairs.6
state_dict()An ordered dict: parameters and persistent buffers.16 keys

The dotted name is the path from the root: each attribute name, joined by dots. Sequential and ModuleList name their children 0, 1, 2. So the convolution weight in TinyNet is backbone.0.weight. These names are the keys of the checkpoint. Rename an attribute and every old checkpoint stops matching (Chapter 4).

Walk the tree

Pick a walk and press Play, or drag the step slider. The cursor visits the tree depth first. The list on the right fills with what the walk yields, in order.

13

Names, order and counts are what PyTorch 2.10 prints for TinyNet. state_dict() visits a module's own parameters, then its buffers, then its children.

Count it three ways

The total size of a model is the sum of the sizes of its parameters. Each parameter's size is the product of its shape.

numel(p)
The number of elements in parameter p: the product of its shape. A (16, 3, 3, 3) weight has 432.
N
The model's parameter count, summed over model.parameters(). Shared parameters count once.

Worked, TinyNet by hand: conv 16·3·3·3 + 16 = 448; BN 16 + 16 = 32; conv 32·16·3·3 + 32 = 4,640; BN 32 + 32 = 64; linear 32·10 + 10 = 330. Sum: 5,514.

n_all   = sum(p.numel() for p in model.parameters())                     # 5514
n_train = sum(p.numel() for p in model.parameters() if p.requires_grad)
per_top = {name: sum(p.numel() for p in m.parameters())
           for name, m in model.named_children()}                   # {'backbone': 5184, 'head': 330}
conv = model.get_submodule("backbone.0")                            # reach a node by its dotted name

For the third way, libraries such as torchinfo print a table of every layer with its output shape and count. Use the one-liner to check the library, and the hand count to check your understanding.

In TinyNet, which call yields exactly two items?

Chapter 4 How it works Bug to spot

Save and load: state_dict

A checkpoint is a dictionary from dotted names to tensors. Loading is name matching. Learn the rules, the errors, and the four repairs you will need

A state_dict is an ordered dictionary. Its keys are the dotted names of every parameter and persistent buffer. Its values are the tensors. That is all a checkpoint is. torch.save(model.state_dict(), "ckpt.pt") writes it with Python's pickle. The class, the code and the forward pass are not saved.

The values share memory with the live model and carry no gradient history. Change the model after sd = model.state_dict() and sd changes too. To keep a snapshot in memory, copy it: {k: v.clone() for k, v in sd.items()}.

Loading goes the other way. model.load_state_dict(sd) walks the model and, for each of its keys, copies the tensor with the same key from sd into place. So loading is matching names. Three things can go wrong:

With the default strict=True, any missing or unexpected key raises a RuntimeError. With strict=False, they are allowed; the call returns them in a named tuple. A size mismatch raises an error either way.

The checkpoint matcher

Pick where the checkpoint came from. Then switch on repairs until the model loads. Left: the checkpoint's keys after your repairs. Right: the model's keys. Green lines match.

The model is NewNet from the lab below: encoder (Linear 20→64, ReLU, Linear 64→32, LayerNorm 32) and head (Linear 32→10). Error texts are PyTorch 2.10's.

The four repairs

Strip a prefix. nn.DataParallel and DistributedDataParallel wrap your model as an attribute called module. Save the wrapper's state_dict and every key starts with module.. Save model.module.state_dict() instead, or strip the prefix when loading. torch.compile adds _orig_mod. the same way.

Rename. You renamed self.fc to self.head. Map old keys to new ones with a small function. Never rename by hand-editing tensors.

Load part of it. You want the pretrained encoder but a new head for a new number of classes. Keep only the encoder keys and load with strict=False. Then check that the missing keys are exactly the new head.

Check the result. strict=False hides mistakes. Always read what it returns:

sd = torch.load("ckpt.pt", map_location="cpu", weights_only=True)
sd = {k.removeprefix("module."): v for k, v in sd.items()}              # repair 1
sd = {k.replace("fc.", "head.", 1) if k.startswith("fc.") else k: v
      for k, v in sd.items()}                                          # repair 2
enc = {k: v for k, v in sd.items() if k.startswith("encoder.")}    # repair 3
res = model.load_state_dict(enc, strict=False)
assert set(res.missing_keys) == {"head.weight", "head.bias"}, res          # repair 4: check
assert not res.unexpected_keys, res
You load a 10-class checkpoint into a 5-class model with strict=False. What happens?

Chapter 5 How it works Bug to spot

train() and eval()

One flag on every module. Most layers ignore it. Dropout and BatchNorm change their maths. Forget it and your test numbers are wrong

Every module has a flag, self.training. It starts as True. model.train() sets it to True on the model and every module below it. model.eval() sets it to False. That is all they do. They do not stop gradients, and they do not freeze anything.

Most layers ignore the flag. A Linear or a Conv2d computes the same thing in both modes. Two common layers do not:

Why the switch? At test time you want the answer for one example to depend only on that example. Batch statistics would mix in the other examples in the batch. Random dropout would make the same input give different answers.

μ, σ2
Training mode: the batch's mean and variance, per channel. Eval mode: the running mean and running variance.
ε
A small number, 1e-5 by default, so we never divide by zero.
γ, β
A learned scale and shift per channel. They are the Parameters weight and bias; they start at 1 and 0.
m
The momentum, 0.1 by default. Each batch moves the running average 10% of the way.
r
The buffer running_mean, which starts at 0. running_var starts at 1 and is updated the same way, with the unbiased batch variance.
y
The output, the same shape as x.

Worked, data with true mean 5: after one batch r = 0.9 × 0 + 0.1 × 5 = 0.5. After k batches r ≈ 5 × (1 − 0.9k): k = 10 gives 3.26, k = 30 gives 4.79. A model evaluated after 3 batches would subtract 1.36 instead of 5.

BatchNorm and Dropout, two modes

Inputs come from one channel with mean 5 and spread 2. Switch the mode. Shrink the batch. Let the running stats see few or many batches. Then read the output's mean and spread.

16
30

One channel of nn.BatchNorm1d(C) with its defaults (eps 1e-5, momentum 0.1, γ = 1, β = 0), and nn.Dropout(p=0.2) on 16 ones. Random draws are seeded; values are illustrative.

What does model.eval() change?

Chapter 6 Build it Bug to spot

Freeze, and split the learning rate

Freezing stops a weight from changing. eval() changes how a layer computes. They are different switches. Then give each part of the model its own learning rate

You have a pretrained backbone and a new head (see Backbone + Head). You want the head to learn and the backbone to stay as it is. That is freezing. In PyTorch it is one call:

model.backbone.requires_grad_(False)     # every Parameter below backbone: requires_grad = False

Autograd then makes no gradient for those weights. Their .grad stays None. PyTorch's optimizers skip a parameter whose .grad is None, weight decay included. If the frozen part sits at the bottom of the network, backward also stops early, which saves time and memory.

Freezing does not touch the training flag. A frozen BatchNorm in training mode still updates its running mean and variance from your batches. So the "frozen" backbone still drifts. The two switches are separate:

train() modeeval() mode
requires_grad = TrueNormal training: weights learn, BN stats update, dropout on.Weights learn, but BN uses fixed stats and dropout is off. Used to fine-tune with frozen BN stats.
requires_grad = FalseWeights fixed, but BN stats still drift and dropout is on. Usually a bug.Truly frozen. The safe way to freeze a backbone.

Parameter groups

Sometimes you want the backbone to move, but slowly. A head starts from random weights and must move far. A pretrained backbone starts near a good answer and should move a little. An optimizer takes a list of parameter groups. Each group is a dictionary with its own params and its own settings. Any setting a group leaves out comes from the defaults you pass after the list.

opt = torch.optim.AdamW([
    {"params": model.backbone.parameters(), "lr": 1e-4},   # slow
    {"params": model.head.parameters()},                    # uses the default lr below
], lr=1e-3, weight_decay=0.01)
for g in opt.param_groups: print(g["lr"], len(g["params"]))       # 0.0001 ..., 0.001 ...
θ
One parameter tensor, in group g.
ηg
That group's learning rate. Schedulers scale every group's rate by the same factor, so the ratio between groups stays.
u(θ)
The optimizer's step direction: the gradient for SGD, the momentum-scaled gradient for Adam.

Worked, plain SGD, gradient 0.5 on one backbone weight and one head weight: backbone 1e-4 × 0.5 = 0.00005; head 1e-3 × 0.5 = 0.0005. Same gradient, ten times the step.

Freeze, fine-tune, or split the rate?

A small backbone was "pretrained" for an old task. Now a new head learns a new task. Pick a strategy and a head learning rate. Solid green: new-task accuracy. Dashed grey: old-task accuracy, read through the same backbone. If the backbone moves too much, the old task is forgotten.

0.6

Backbone: 2 → 8 tanh units with fixed random weights; the old task is a straight boundary read by a logistic head fitted once. New task: the sign of x0x1 on 400 points. 250 full-batch SGD steps with momentum 0.9, trained in your browser. Numbers are illustrative.

You call backbone.requires_grad_(False) but leave the model in train mode. The backbone has BatchNorm. What still changes during training?

Chapter 7 How it works Build it

Initialisation

Every layer starts from random numbers chosen by a rule. See what the defaults are, why a deep ReLU stack with them goes quiet, and how to re-initialise a whole model with apply

A new layer is not empty. Its constructor calls reset_parameters(), which fills the weights with random numbers. The range matters. Too small, and the signal fades layer by layer until the last layers see nothing. Too large, and it explodes.

PyTorch's defaults:

Why does the size of the signal change? Take one output of a linear layer: a sum of fan_in products w·x. If the weights and inputs are independent with mean zero, the variances add:

n
fan_in: how many products are summed.
Var(w)
The spread of the initial weights. A uniform range ±b has variance b2/3.
E[x2]
The mean square of the layer's input: the size of the signal coming in.
Var(y)
The spread of the output before the ReLU. ReLU zeroes the negative half, which halves the mean square.

Worked, default init: Var(w) = (1/√n)2/3 = 1/(3n), so each Linear + ReLU block multiplies the mean square by n · 1/(3n) · 1/2 = 1/6. The size (its square root) shrinks by √(1/6) = 0.41 per block; after 5 blocks, 0.415 ≈ 0.012. Kaiming init uses Var(w) = 2/n: the factor is n · 2/n · 1/2 = 1.

Push a signal through a deep stack

A batch of random inputs runs through a stack of Linear layers and activations, built live with each rule. The line is the signal's size after each layer, on a log scale. Flat is healthy.

12

Width 64, batch 64, inputs N(0, 1). "PyTorch default" also draws biases in ±1/√64, which sets a floor near 0.05; the other rules use zero bias. Computed live; numbers vary a little with the seed.

So the default is not tuned for deep ReLU stacks. It rarely hurts in practice, because modern networks add normalisation layers and skip connections that reset the scale (see Skip connections). But for a plain deep MLP, or a layer you wrote yourself, choose the init on purpose. The Initialization lesson derives each rule.

Re-initialise with apply

model.apply(fn) calls fn(m) on every submodule, children first, then the module itself. Inside fn, check the type and call the nn.init functions. They end in an underscore because they work in place, and they run under no_grad for you.

def init_weights(m):
    if isinstance(m, (nn.Linear, nn.Conv2d)):
        nn.init.kaiming_normal_(m.weight, nonlinearity="relu")    # std = sqrt(2 / fan_in)
        if m.bias is not None:
            nn.init.zeros_(m.bias)
    elif isinstance(m, nn.LayerNorm):
        nn.init.ones_(m.weight); nn.init.zeros_(m.bias)

torch.manual_seed(0)              # seed BEFORE building, so the init is repeatable
model = Net()
model.apply(init_weights)
With PyTorch's default init, by about how much does each Linear + ReLU block shrink the signal's size (RMS)?

Chapter 8 Build it Bug to spot

Hooks: look inside without editing

Attach a function to any layer and it runs every time that layer runs. Use it to read features, check for NaNs, or change an input. Then remove it

You want the output of layer 3 of a pretrained model. You could copy the model's code and add a return. A hook is cleaner. It is a function you attach to a module. Module.__call__ runs it around forward, every time.

Each call returns a handle. handle.remove() takes the hook off. Without it, the hook stays for the life of the module.

feats = {}
def save_to(name):
    def hook(module, args, output):
        feats[name] = output.detach()                # detach: keep the values, drop the graph
    return hook

handles = [model.get_submodule(n).register_forward_hook(save_to(n))
           for n in ["backbone.2", "backbone.5"]]
with torch.no_grad():
    model(images)                                    # feats now holds both feature maps
for h in handles: h.remove()

Place a hook

Pick a layer and a hook kind. The hook sees the shape at its edge of the layer, and for image layers we draw what it captured. Then let the hook append to a list over many passes, and watch the memory it holds.

1

The network of the lab below, on a batch of 64 images of 24 × 24, in float32. A stored tensor that is not detached keeps its autograd graph alive, and with it the saved activations of every layer before it. Memory counts those tensors only.

Your forward hook returns output * 0. What happens?

Chapter 9 Build it

Your own layer, and the containers

Write a layer the way PyTorch's own layers are written, with a clean printout. Then pick the right container for a stack of layers

PyTorch's own layers all follow one pattern. Copy it and your layers will save, load, print and initialise like the built-in ones.

class ScaleShift(nn.Module):
    """y = x * gamma + beta, one gamma and beta per feature. x: (..., dim)"""
    def __init__(self, dim, init_scale=1.0):
        super().__init__()                                   # 1. registries first
        self.dim, self.init_scale = dim, init_scale          # 2. plain config, for the repr
        self.gamma = nn.Parameter(torch.empty(dim))          # 3. register, then fill
        self.beta = nn.Parameter(torch.empty(dim))
        self.reset_parameters()

    def reset_parameters(self):                          # 4. one place for the init
        nn.init.constant_(self.gamma, self.init_scale)
        nn.init.zeros_(self.beta)

    def forward(self, x):                                  # 5. shapes in the docstring
        return x * self.gamma + self.beta

    def extra_repr(self):                                 # 6. what print() shows
        return f"{self.dim}, init_scale={self.init_scale}"

extra_repr returns the text inside the brackets when you print the module. Without it, print(model) shows ScaleShift(), which tells a reader nothing. With it, the line reads ScaleShift(64, init_scale=1.0), like Linear(in_features=64, …).

To hold many layers, pick a container. All three are modules, so __setattr__ registers them and everything inside.

Pick a container

Each choice holds the same two layers: a ScaleShift(4) and a Linear(4, 2), stored as self.body. Read the printout, the parameter names, and the result of calling it. Then switch extra_repr off.

Printouts and error texts are what PyTorch 2.10 gives for these modules.

You need a stack of blocks where each block also receives the original input x. Which container fits?

Chapter 10 Build it

Stateless calls, and what a model costs

Run a module with weights it does not own, for ensembles and meta-learning. Then count the memory a model needs to train

A module bundles two things: code (its forward) and state (its parameters and buffers). Sometimes you want the code with different state. You may have five sets of trained weights for the same architecture and want all five answers. Or you may want to try a step of training, see how the loss changes, and backpropagate through that step.

torch.func.functional_call(module, params, args) does this. You pass a dictionary from dotted names to tensors. For that one call, the module uses your tensors in place of its own. Its own parameters do not change. Names you leave out use the module's own tensors.

from torch.func import functional_call
params = {k: v.detach() for k, v in other_model.named_parameters()}
y = functional_call(model, params, (x,))           # model's code, other_model's weights

Ensembles. Keep one module object as a skeleton, and a list of weight dictionaries. Average the answers. The lab builds this.

Meta-learning. MAML-style training takes one inner gradient step, then measures the loss after that step. The "fast weights" are ordinary tensors computed from the real parameters. functional_call runs the model with them, and the gradient flows back to the real parameters (see meta-RL):

p = dict(model.named_parameters())
inner = F.mse_loss(functional_call(model, p, (x_support,)), y_support)
g = torch.autograd.grad(inner, list(p.values()), create_graph=True)   # keep the graph
fast = {k: v - 0.1 * gk for (k, v), gk in zip(p.items(), g)}      # one inner SGD step
outer = F.mse_loss(functional_call(model, fast, (x_query,)), y_query)
outer.backward()                                    # grads reach model's real parameters

What a model costs to train

A parameter costs memory several times over during training. The weight itself. Its gradient, the same size. And the optimizer's state: SGD with momentum keeps one extra tensor per parameter, and Adam keeps two (exp_avg and exp_avg_sq).

N
The number of parameters (Chapter 3). For bg and bo, count only the trainable ones.
bw
Bytes per weight: 4 for float32, 2 for bfloat16.
bg
Bytes per gradient: the same dtype as the weight.
bo
Optimizer state: 0 for plain SGD, 4 for momentum, 8 for Adam's two float32 moments. Mixed precision adds a 4-byte float32 master copy.
M
The memory for model state, before any activations.

Worked: a 7-billion-parameter model with float32 Adam: 7×109 × (4 + 4 + 8) = 112 GB. Mixed-precision Adam: 2 + 2 + 4 + 8 = 16 bytes, also 112 GB. Inference in bfloat16: 2 bytes, 14 GB. TinyNet in float32 for inference: 5,514 × 4 = 22 KB.

What it costs to train

Set the size, the precision and the optimizer. Then freeze most of the model and watch the gradient and optimizer parts shrink.

1.0 B
100%

Model state only; activations, temporary buffers and fragmentation come on top and are often larger. The 16 bytes per parameter of mixed-precision Adam is the count used in the ZeRO paper (Rajbhandari et al., 2020, §3.1). GPU sizes are common memory sizes, for scale.

After functional_call(model, other_params, (x,)), what are model's own parameters?

Chapter 11

Connections

Turn a symptom into its likely cause, keep the one-screen cheat sheet, know the limits, and go on to the next part of the series

Most nn.Module bugs show up as a strange symptom far from the cause. Pick a symptom to see the usual cause, the one-line check, and the fix.

From symptom to cause

Each card links back to the chapter that explains it.

The usual cause, not the only one. Always confirm with the check before you change code.

The cheat sheet

JobIn one line
Registration__setattr__ files Parameters, Modules and known buffers; everything else is invisible. super().__init__() first.
Three kindsParameter: trained, saved, moved. Buffer: saved, moved. persistent=False: moved only. Plain tensor: none.
ContainersSequential runs in order; ModuleList / ModuleDict register, you loop. Never a Python list.
Names and countsDotted paths, depth first. sum(p.numel() for p in model.parameters()); add if p.requires_grad for trainable.
Checkpointsstate_dict = names → tensors. Strip module., rename, filter; strict=False only with asserts on the result.
Modestrain()/eval() flip training; only Dropout, BatchNorm and similar care. Not the same as no_grad.
Freezingrequires_grad_(False) + eval() on the frozen part; rebuild the optimizer; param groups for per-part LRs.
InitDefault ±1/√fan_in shrinks deep ReLU stacks by 0.41 per block; Kaiming sqrt(2/fan_in) keeps size; model.apply(fn).
HooksPre: (module, args). Forward: (module, args, output). Detach what you keep; handle.remove().
Statelessfunctional_call(model, params, args): model's code, your tensors, for one call.
MemoryTrain with Adam: about 16 bytes per trainable parameter, plus activations.

Honest limits

Where this sits, and where to go next

The takeaway. A module is a tree with three registries at every node. Every method you call on a model walks that tree. So when something is wrong, ask three questions: did the module register it, does the walk reach it, and is the training flag what you think it is?
A fine-tuned model scores well in your notebook. Reloaded in a server, it scores near chance. The weights file loaded with strict=False. What do you check first?

Now press Present or Teach and explain, out loud and from memory, why a layer stored in a Python list is never trained or saved. Then go back to the hero, freeze the backbone, switch to eval, and say which numbers changed and why.

nn.Module Inside Out, part 3 of PyTorch, Zero to Expert in The Builder's Toolkit. Every default, name, shape, count and error message on this page was checked against PyTorch 2.10.0. Sources: the PyTorch source code (torch/nn/modules/module.py, container.py, batchnorm.py, linear.py, torch/nn/init.py, torch/_functorch/functional_call.py) and documentation (torch.nn.Module, Saving and loading models); S. Ioffe, C. Szegedy, "Batch Normalization" (ICML 2015, arXiv); N. Srivastava et al., "Dropout" (JMLR 2014); K. He et al., "Delving Deep into Rectifiers" (Kaiming init, ICCV 2015, arXiv); X. Glorot, Y. Bengio, "Understanding the difficulty of training deep feedforward neural networks" (AISTATS 2010); C. Finn, P. Abbeel, S. Levine, "Model-Agnostic Meta-Learning" (ICML 2017, arXiv); S. Rajbhandari et al., "ZeRO: Memory Optimizations Toward Training Trillion Parameter Models" (SC 2020, arXiv).
Back to Gleams