Every PyTorch model is a tree of modules. Each module keeps three lists: its parameters, its buffers and its child modules. Training, saving, loading, freezing and moving to a GPU all walk that tree. Learn the tree, and most model bugs become easy to see.
Learn how a module records what you assign to it, what goes into a checkpoint, what train() and eval() change, how to freeze part of a model, how to look inside with hooks, and how to run a model with weights it does not own. Eight Code Labs run real PyTorch code in your browser.
Click any row of the tree. Freeze a branch, or switch the model to eval mode. Watch what trains, what is saved, and what changes.
You need tensors and autograd: see Tensors and Autograd. We build everything else from zero.
TinyNet is a real model: two convolution blocks with BatchNorm, then a dropout and linear head for 10 classes. Every name, shape and count here is what PyTorch 2.10 prints for it. Click a row to select it; click its arrow to fold it.
Chapter 0 Bug to spot
A model with a hidden mistake trains, the loss falls, and the saved checkpoint is still broken. Find out why, then see the map of this lesson
You write a small network. It has an input layer, three hidden layers and an output layer. You keep the hidden layers in a Python list, because you want to loop over them. You train it. The loss falls. You save the weights and go home.
The next day you load the weights into a fresh copy of the model. Its predictions are nonsense. Nothing raised an error. The loss curve looked healthy. What went wrong?
PyTorch never saw the hidden layers. A plain Python list is invisible to the model. So the optimizer did not train those layers, and the checkpoint did not save them. The fresh copy got new random hidden layers. Try the three ways to store the layers below.
Pick how the hidden layers are stored. The network trains live in your browser. The left side shows what the model registered. The right side shows the training loss, then the loss of a copy reloaded from the checkpoint.
A 2 → 16 → 16 → 16 → 1 tanh network fits y = sin(3x0) cos(2x1) on 256 points with Adam (lr 0.01, 300 steps), trained in JavaScript with the same maths as PyTorch. "Reload" builds a fresh model and copies in only what the state_dict holds. Loss values are illustrative.
The loss curve is the trap. Gradients still flow through the hidden layers, so the input and output layers learn to work around them. Training looks fine. Only the reload shows the damage. This is why experts check what a model registered, not just whether the loss falls.
Each chapter takes one job of nn.Module, shows how it works inside, and names the bugs experts look for.
__setattr__ and the three registries. Lab: write a layer from scratch.children, modules, named_parameters and the dotted names.state_dict, strict loading, missing and unexpected keys. Lab: load a renamed checkpoint.requires_grad_ and parameter groups. Lab: freeze a backbone, give the head 10× the LR.apply. Lab: re-initialise a network.extra_repr, Sequential, ModuleList, ModuleDict.functional_call, ensembles, meta-learning and memory. Lab: an ensemble from one module.This is part 3 of an eleven-part series in The Builder's Toolkit.
Chapter 1 How it works
Every self.x = … inside a module goes through one method that sorts the value into a registry. Watch it sort, then write a layer from scratch
Think of a module as a desk with labelled trays. When you write self.fc = nn.Linear(4, 2), you do not just set an attribute. You hand the value to a clerk. The clerk looks at it and drops it into the right tray. Everything PyTorch does later reads the trays, not your attributes.
The clerk is nn.Module.__setattr__. Python calls it for every assignment of the form self.name = value. It checks the value's type and sorts it:
nn.Parameter goes into self._parameters. A Parameter is a tensor that the module owns and that should be trained. It has requires_grad=True by default.nn.Module goes into self._modules. This is how a model becomes a tree: a child module is just another tray entry.self._buffers. A buffer is a tensor the module saves and moves but never trains, such as BatchNorm's running mean. You create one with self.register_buffer(name, tensor).That is why super().__init__() must come first in your __init__. It creates the empty trays. Assign a layer before it, and PyTorch raises AttributeError: cannot assign module before Module.__init__() call.
Pick a line of __init__. The lit path shows the checks __setattr__ makes. The value lands in a tray. Only the first three trays are seen by parameters(), state_dict() and .to().
The checks follow torch/nn/modules/module.py. register_buffer is a method call, not an assignment, so it writes the buffer tray directly. Once a name is a buffer, a later self.mean = tensor goes through __setattr__ and updates that buffer.
A linear layer computes y = x WT + b. Its weight W has one row per output and one column per input. Its bias b has one number per output. So its size is easy to count:
weight.numel() + bias.numel().Worked, a 784 → 256 → 10 network: 784 × 256 + 256 = 200,960; 256 × 10 + 10 = 2,570; total 203,530. The ReLU between them owns nothing.
Here is a linear layer written from scratch. Read it line by line; then build it in the lab.
import math, torch, torch.nn as nn class MyLinear(nn.Module): def __init__(self, n_in, n_out): super().__init__() # creates the empty registries self.n_in, self.n_out = n_in, n_out # plain ints: kept, not registered bound = 1 / math.sqrt(n_in) # PyTorch's default range self.weight = nn.Parameter(torch.empty(n_out, n_in).uniform_(-bound, bound)) self.bias = nn.Parameter(torch.empty(n_out).uniform_(-bound, bound)) def forward(self, x): # x: (B, n_in) -> (B, n_out) return x @ self.weight.T + self.bias
You never call forward yourself. You call the module: layer(x). That goes through Module.__call__, which runs any hooks (Chapter 8) and then forward. Calling layer.forward(x) directly skips the hooks.
__init__ you write self.alpha = torch.ones(3, requires_grad=True). What happens?Chapter 2 How it works Bug to spot
Five everyday operations treat each kind of attribute differently. Learn the grid once, then fix the Python-list bug for real
A module holds three kinds of tensor. They look alike when you print them. They behave very differently when you train, save or move the model.
A buffer can also be non-persistent: register_buffer(name, t, persistent=False). It moves with the model but is left out of the checkpoint. Use it for things you can rebuild, such as a cached table of sines for positional encodings.
Pick an operation. Each row is one way to hold a tensor or a layer. A filled mark means the operation reaches it.
Each mark was checked by running the operation in PyTorch 2.10. "backward()" means the tensor gets a .grad. "opt.step()" assumes the optimizer was built from model.parameters().
Look at the "layer in a Python list" row under backward(). Its weights do get gradients, because they are real Parameters used in the forward pass. But the optimizer was built from model.parameters(), which never yields them. So the gradients pile up and nothing ever uses them. That is the bug from Chapter 0. .double() also skips them, so the model ends up with mixed dtypes and crashes in the forward pass.
The fix is one word: nn.ModuleList. It is a list that is also a module, so __setattr__ files it under _modules, and its items get names blocks.0, blocks.1 and so on.
Chapter 3 How it works
Every walk over a model visits the same tree in the same order. Learn the walks, the dotted names they produce, and how to count what a model owns
Since a model is a tree, most of the nn.Module API is a walk over it. The walk is always the same: visit a module, then each child in the order you assigned them, depth first. What changes is what each walk yields.
| Call | Yields | For TinyNet |
|---|---|---|
children() | Direct child modules only. | 2: backbone, head |
modules() | Every module, the root first. | 13 |
named_parameters() | (dotted name, Parameter) pairs. | 10 tensors, 5,514 numbers |
named_buffers() | (dotted name, buffer) pairs. | 6 |
state_dict() | An ordered dict: parameters and persistent buffers. | 16 keys |
The dotted name is the path from the root: each attribute name, joined by dots. Sequential and ModuleList name their children 0, 1, 2. So the convolution weight in TinyNet is backbone.0.weight. These names are the keys of the checkpoint. Rename an attribute and every old checkpoint stops matching (Chapter 4).
Pick a walk and press Play, or drag the step slider. The cursor visits the tree depth first. The list on the right fills with what the walk yields, in order.
Names, order and counts are what PyTorch 2.10 prints for TinyNet. state_dict() visits a module's own parameters, then its buffers, then its children.
The total size of a model is the sum of the sizes of its parameters. Each parameter's size is the product of its shape.
model.parameters(). Shared parameters count once.Worked, TinyNet by hand: conv 16·3·3·3 + 16 = 448; BN 16 + 16 = 32; conv 32·16·3·3 + 32 = 4,640; BN 32 + 32 = 64; linear 32·10 + 10 = 330. Sum: 5,514.
n_all = sum(p.numel() for p in model.parameters()) # 5514 n_train = sum(p.numel() for p in model.parameters() if p.requires_grad) per_top = {name: sum(p.numel() for p in m.parameters()) for name, m in model.named_children()} # {'backbone': 5184, 'head': 330} conv = model.get_submodule("backbone.0") # reach a node by its dotted name
For the third way, libraries such as torchinfo print a table of every layer with its output shape and count. Use the one-liner to check the library, and the hand count to check your understanding.
Chapter 4 How it works Bug to spot
A checkpoint is a dictionary from dotted names to tensors. Loading is name matching. Learn the rules, the errors, and the four repairs you will need
A state_dict is an ordered dictionary. Its keys are the dotted names of every parameter and persistent buffer. Its values are the tensors. That is all a checkpoint is. torch.save(model.state_dict(), "ckpt.pt") writes it with Python's pickle. The class, the code and the forward pass are not saved.
The values share memory with the live model and carry no gradient history. Change the model after sd = model.state_dict() and sd changes too. To keep a snapshot in memory, copy it: {k: v.clone() for k, v in sd.items()}.
Loading goes the other way. model.load_state_dict(sd) walks the model and, for each of its keys, copies the tensor with the same key from sd into place. So loading is matching names. Three things can go wrong:
With the default strict=True, any missing or unexpected key raises a RuntimeError. With strict=False, they are allowed; the call returns them in a named tuple. A size mismatch raises an error either way.
Pick where the checkpoint came from. Then switch on repairs until the model loads. Left: the checkpoint's keys after your repairs. Right: the model's keys. Green lines match.
The model is NewNet from the lab below: encoder (Linear 20→64, ReLU, Linear 64→32, LayerNorm 32) and head (Linear 32→10). Error texts are PyTorch 2.10's.
Strip a prefix. nn.DataParallel and DistributedDataParallel wrap your model as an attribute called module. Save the wrapper's state_dict and every key starts with module.. Save model.module.state_dict() instead, or strip the prefix when loading. torch.compile adds _orig_mod. the same way.
Rename. You renamed self.fc to self.head. Map old keys to new ones with a small function. Never rename by hand-editing tensors.
Load part of it. You want the pretrained encoder but a new head for a new number of classes. Keep only the encoder keys and load with strict=False. Then check that the missing keys are exactly the new head.
Check the result. strict=False hides mistakes. Always read what it returns:
sd = torch.load("ckpt.pt", map_location="cpu", weights_only=True) sd = {k.removeprefix("module."): v for k, v in sd.items()} # repair 1 sd = {k.replace("fc.", "head.", 1) if k.startswith("fc.") else k: v for k, v in sd.items()} # repair 2 enc = {k: v for k, v in sd.items() if k.startswith("encoder.")} # repair 3 res = model.load_state_dict(enc, strict=False) assert set(res.missing_keys) == {"head.weight", "head.bias"}, res # repair 4: check assert not res.unexpected_keys, res
strict=False. What happens?Chapter 5 How it works Bug to spot
One flag on every module. Most layers ignore it. Dropout and BatchNorm change their maths. Forget it and your test numbers are wrong
Every module has a flag, self.training. It starts as True. model.train() sets it to True on the model and every module below it. model.eval() sets it to False. That is all they do. They do not stop gradients, and they do not freeze anything.
Most layers ignore the flag. A Linear or a Conv2d computes the same thing in both modes. Two common layers do not:
Why the switch? At test time you want the answer for one example to depend only on that example. Batch statistics would mix in the other examples in the batch. Random dropout would make the same input give different answers.
weight and bias; they start at 1 and 0.momentum, 0.1 by default. Each batch moves the running average 10% of the way.running_mean, which starts at 0. running_var starts at 1 and is updated the same way, with the unbiased batch variance.Worked, data with true mean 5: after one batch r = 0.9 × 0 + 0.1 × 5 = 0.5. After k batches r ≈ 5 × (1 − 0.9k): k = 10 gives 3.26, k = 30 gives 4.79. A model evaluated after 3 batches would subtract 1.36 instead of 5.
Inputs come from one channel with mean 5 and spread 2. Switch the mode. Shrink the batch. Let the running stats see few or many batches. Then read the output's mean and spread.
One channel of nn.BatchNorm1d(C) with its defaults (eps 1e-5, momentum 0.1, γ = 1, β = 0), and nn.Dropout(p=0.2) on 16 ones. Random draws are seeded; values are illustrative.
model.eval() change?Chapter 6 Build it Bug to spot
Freezing stops a weight from changing. eval() changes how a layer computes. They are different switches. Then give each part of the model its own learning rate
You have a pretrained backbone and a new head (see Backbone + Head). You want the head to learn and the backbone to stay as it is. That is freezing. In PyTorch it is one call:
model.backbone.requires_grad_(False) # every Parameter below backbone: requires_grad = False
Autograd then makes no gradient for those weights. Their .grad stays None. PyTorch's optimizers skip a parameter whose .grad is None, weight decay included. If the frozen part sits at the bottom of the network, backward also stops early, which saves time and memory.
Freezing does not touch the training flag. A frozen BatchNorm in training mode still updates its running mean and variance from your batches. So the "frozen" backbone still drifts. The two switches are separate:
| train() mode | eval() mode | |
|---|---|---|
| requires_grad = True | Normal training: weights learn, BN stats update, dropout on. | Weights learn, but BN uses fixed stats and dropout is off. Used to fine-tune with frozen BN stats. |
| requires_grad = False | Weights fixed, but BN stats still drift and dropout is on. Usually a bug. | Truly frozen. The safe way to freeze a backbone. |
Sometimes you want the backbone to move, but slowly. A head starts from random weights and must move far. A pretrained backbone starts near a good answer and should move a little. An optimizer takes a list of parameter groups. Each group is a dictionary with its own params and its own settings. Any setting a group leaves out comes from the defaults you pass after the list.
opt = torch.optim.AdamW([
{"params": model.backbone.parameters(), "lr": 1e-4}, # slow
{"params": model.head.parameters()}, # uses the default lr below
], lr=1e-3, weight_decay=0.01)
for g in opt.param_groups: print(g["lr"], len(g["params"])) # 0.0001 ..., 0.001 ...
Worked, plain SGD, gradient 0.5 on one backbone weight and one head weight: backbone 1e-4 × 0.5 = 0.00005; head 1e-3 × 0.5 = 0.0005. Same gradient, ten times the step.
A small backbone was "pretrained" for an old task. Now a new head learns a new task. Pick a strategy and a head learning rate. Solid green: new-task accuracy. Dashed grey: old-task accuracy, read through the same backbone. If the backbone moves too much, the old task is forgotten.
Backbone: 2 → 8 tanh units with fixed random weights; the old task is a straight boundary read by a logistic head fitted once. New task: the sign of x0x1 on 400 points. 250 full-batch SGD steps with momentum 0.9, trained in your browser. Numbers are illustrative.
backbone.requires_grad_(False) but leave the model in train mode. The backbone has BatchNorm. What still changes during training?Chapter 7 How it works Build it
Every layer starts from random numbers chosen by a rule. See what the defaults are, why a deep ReLU stack with them goes quiet, and how to re-initialise a whole model with apply
A new layer is not empty. Its constructor calls reset_parameters(), which fills the weights with random numbers. The range matters. Too small, and the signal fades layer by layer until the last layers see nothing. Too large, and it explodes.
PyTorch's defaults:
Linear and Conv*d: weights uniform in ±1/√fan_in, where fan_in is the number of inputs to one output (nin for Linear, channels × kernel area for a convolution). Biases use the same range. The code calls it kaiming_uniform_(a=√5).Embedding: standard normal, N(0, 1).LayerNorm and BatchNorm: weight 1, bias 0.Why does the size of the signal change? Take one output of a linear layer: a sum of fan_in products w·x. If the weights and inputs are independent with mean zero, the variances add:
Worked, default init: Var(w) = (1/√n)2/3 = 1/(3n), so each Linear + ReLU block multiplies the mean square by n · 1/(3n) · 1/2 = 1/6. The size (its square root) shrinks by √(1/6) = 0.41 per block; after 5 blocks, 0.415 ≈ 0.012. Kaiming init uses Var(w) = 2/n: the factor is n · 2/n · 1/2 = 1.
A batch of random inputs runs through a stack of Linear layers and activations, built live with each rule. The line is the signal's size after each layer, on a log scale. Flat is healthy.
Width 64, batch 64, inputs N(0, 1). "PyTorch default" also draws biases in ±1/√64, which sets a floor near 0.05; the other rules use zero bias. Computed live; numbers vary a little with the seed.
So the default is not tuned for deep ReLU stacks. It rarely hurts in practice, because modern networks add normalisation layers and skip connections that reset the scale (see Skip connections). But for a plain deep MLP, or a layer you wrote yourself, choose the init on purpose. The Initialization lesson derives each rule.
model.apply(fn) calls fn(m) on every submodule, children first, then the module itself. Inside fn, check the type and call the nn.init functions. They end in an underscore because they work in place, and they run under no_grad for you.
def init_weights(m): if isinstance(m, (nn.Linear, nn.Conv2d)): nn.init.kaiming_normal_(m.weight, nonlinearity="relu") # std = sqrt(2 / fan_in) if m.bias is not None: nn.init.zeros_(m.bias) elif isinstance(m, nn.LayerNorm): nn.init.ones_(m.weight); nn.init.zeros_(m.bias) torch.manual_seed(0) # seed BEFORE building, so the init is repeatable model = Net() model.apply(init_weights)
Chapter 8 Build it Bug to spot
Attach a function to any layer and it runs every time that layer runs. Use it to read features, check for NaNs, or change an input. Then remove it
You want the output of layer 3 of a pretrained model. You could copy the model's code and add a return. A hook is cleaner. It is a function you attach to a module. Module.__call__ runs it around forward, every time.
register_forward_pre_hook(fn): fn(module, args) runs before forward. args is the tuple of positional inputs. If fn returns something, it replaces the inputs.register_forward_hook(fn): fn(module, args, output) runs after forward. If fn returns something, it replaces the output.register_full_backward_hook(fn): fn(module, grad_input, grad_output) runs during backward. Use it to watch gradients.Each call returns a handle. handle.remove() takes the hook off. Without it, the hook stays for the life of the module.
feats = {}
def save_to(name):
def hook(module, args, output):
feats[name] = output.detach() # detach: keep the values, drop the graph
return hook
handles = [model.get_submodule(n).register_forward_hook(save_to(n))
for n in ["backbone.2", "backbone.5"]]
with torch.no_grad():
model(images) # feats now holds both feature maps
for h in handles: h.remove()
Pick a layer and a hook kind. The hook sees the shape at its edge of the layer, and for image layers we draw what it captured. Then let the hook append to a list over many passes, and watch the memory it holds.
The network of the lab below, on a batch of 64 images of 24 × 24, in float32. A stored tensor that is not detached keeps its autograd graph alive, and with it the saved activations of every layer before it. Memory counts those tensors only.
output * 0. What happens?Chapter 9 Build it
Write a layer the way PyTorch's own layers are written, with a clean printout. Then pick the right container for a stack of layers
PyTorch's own layers all follow one pattern. Copy it and your layers will save, load, print and initialise like the built-in ones.
class ScaleShift(nn.Module): """y = x * gamma + beta, one gamma and beta per feature. x: (..., dim)""" def __init__(self, dim, init_scale=1.0): super().__init__() # 1. registries first self.dim, self.init_scale = dim, init_scale # 2. plain config, for the repr self.gamma = nn.Parameter(torch.empty(dim)) # 3. register, then fill self.beta = nn.Parameter(torch.empty(dim)) self.reset_parameters() def reset_parameters(self): # 4. one place for the init nn.init.constant_(self.gamma, self.init_scale) nn.init.zeros_(self.beta) def forward(self, x): # 5. shapes in the docstring return x * self.gamma + self.beta def extra_repr(self): # 6. what print() shows return f"{self.dim}, init_scale={self.init_scale}"
extra_repr returns the text inside the brackets when you print the module. Without it, print(model) shows ScaleShift(), which tells a reader nothing. With it, the line reads ScaleShift(64, init_scale=1.0), like Linear(in_features=64, …).
To hold many layers, pick a container. All three are modules, so __setattr__ registers them and everything inside.
forward: it calls each child in order, passing the output on. Use it when the data flows straight through.forward. It is a registered list. You write the loop yourself, so you can add skip connections or extra inputs.forward. It is a registered dictionary with string keys. Use it for named branches, such as one head per task.Each choice holds the same two layers: a ScaleShift(4) and a Linear(4, 2), stored as self.body. Read the printout, the parameter names, and the result of calling it. Then switch extra_repr off.
Printouts and error texts are what PyTorch 2.10 gives for these modules.
Chapter 10 Build it
Run a module with weights it does not own, for ensembles and meta-learning. Then count the memory a model needs to train
A module bundles two things: code (its forward) and state (its parameters and buffers). Sometimes you want the code with different state. You may have five sets of trained weights for the same architecture and want all five answers. Or you may want to try a step of training, see how the loss changes, and backpropagate through that step.
torch.func.functional_call(module, params, args) does this. You pass a dictionary from dotted names to tensors. For that one call, the module uses your tensors in place of its own. Its own parameters do not change. Names you leave out use the module's own tensors.
from torch.func import functional_call params = {k: v.detach() for k, v in other_model.named_parameters()} y = functional_call(model, params, (x,)) # model's code, other_model's weights
Ensembles. Keep one module object as a skeleton, and a list of weight dictionaries. Average the answers. The lab builds this.
Meta-learning. MAML-style training takes one inner gradient step, then measures the loss after that step. The "fast weights" are ordinary tensors computed from the real parameters. functional_call runs the model with them, and the gradient flows back to the real parameters (see meta-RL):
p = dict(model.named_parameters()) inner = F.mse_loss(functional_call(model, p, (x_support,)), y_support) g = torch.autograd.grad(inner, list(p.values()), create_graph=True) # keep the graph fast = {k: v - 0.1 * gk for (k, v), gk in zip(p.items(), g)} # one inner SGD step outer = F.mse_loss(functional_call(model, fast, (x_query,)), y_query) outer.backward() # grads reach model's real parameters
A parameter costs memory several times over during training. The weight itself. Its gradient, the same size. And the optimizer's state: SGD with momentum keeps one extra tensor per parameter, and Adam keeps two (exp_avg and exp_avg_sq).
Worked: a 7-billion-parameter model with float32 Adam: 7×109 × (4 + 4 + 8) = 112 GB. Mixed-precision Adam: 2 + 2 + 4 + 8 = 16 bytes, also 112 GB. Inference in bfloat16: 2 bytes, 14 GB. TinyNet in float32 for inference: 5,514 × 4 = 22 KB.
Set the size, the precision and the optimizer. Then freeze most of the model and watch the gradient and optimizer parts shrink.
Model state only; activations, temporary buffers and fragmentation come on top and are often larger. The 16 bytes per parameter of mixed-precision Adam is the count used in the ZeRO paper (Rajbhandari et al., 2020, §3.1). GPU sizes are common memory sizes, for scale.
functional_call(model, other_params, (x,)), what are model's own parameters?Chapter 11
Turn a symptom into its likely cause, keep the one-screen cheat sheet, know the limits, and go on to the next part of the series
Most nn.Module bugs show up as a strange symptom far from the cause. Pick a symptom to see the usual cause, the one-line check, and the fix.
Each card links back to the chapter that explains it.
The usual cause, not the only one. Always confirm with the check before you change code.
| Job | In one line |
|---|---|
| Registration | __setattr__ files Parameters, Modules and known buffers; everything else is invisible. super().__init__() first. |
| Three kinds | Parameter: trained, saved, moved. Buffer: saved, moved. persistent=False: moved only. Plain tensor: none. |
| Containers | Sequential runs in order; ModuleList / ModuleDict register, you loop. Never a Python list. |
| Names and counts | Dotted paths, depth first. sum(p.numel() for p in model.parameters()); add if p.requires_grad for trainable. |
| Checkpoints | state_dict = names → tensors. Strip module., rename, filter; strict=False only with asserts on the result. |
| Modes | train()/eval() flip training; only Dropout, BatchNorm and similar care. Not the same as no_grad. |
| Freezing | requires_grad_(False) + eval() on the frozen part; rebuild the optimizer; param groups for per-part LRs. |
| Init | Default ±1/√fan_in shrinks deep ReLU stacks by 0.41 per block; Kaiming sqrt(2/fan_in) keeps size; model.apply(fn). |
| Hooks | Pre: (module, args). Forward: (module, args, output). Detach what you keep; handle.remove(). |
| Stateless | functional_call(model, params, args): model's code, your tensors, for one call. |
| Memory | Train with Adam: about 16 bytes per trainable parameter, plus activations. |
torch.compile and torch.export trace the module; hooks and Python-side tricks may behave differently under them. Lesson 8 covers compilation.module., torch.compile adds _orig_mod., and FSDP can flatten or shard parameters. Checkpoint code must know which wrapper saved it. Lesson 9 goes inside them..grad comes from).strict=False. What do you check first?Now press Present or Teach and explain, out loud and from memory, why a layer stored in a Python list is never trained or saved. Then go back to the hero, freeze the backbone, switch to eval, and say which numbers changed and why.
torch/nn/modules/module.py, container.py, batchnorm.py, linear.py, torch/nn/init.py, torch/_functorch/functional_call.py) and documentation (torch.nn.Module, Saving and loading models); S. Ioffe, C. Szegedy, "Batch Normalization" (ICML 2015, arXiv); N. Srivastava et al., "Dropout" (JMLR 2014); K. He et al., "Delving Deep into Rectifiers" (Kaiming init, ICCV 2015, arXiv); X. Glorot, Y. Bengio, "Understanding the difficulty of training deep feedforward neural networks" (AISTATS 2010); C. Finn, P. Abbeel, S. Levine, "Model-Agnostic Meta-Learning" (ICML 2017, arXiv); S. Rajbhandari et al., "ZeRO: Memory Optimizations Toward Training Trillion Parameter Models" (SC 2020, arXiv).