You have a pretrained network and a new task. You can train only a new head, a few top layers, a small low-rank patch, or every weight. Each choice changes what you gain on new data, what you lose on shifted data, and what the model forgets.
Probes, adapters, LoRA, prefix and prompt tuning, BitFit and full fine-tuning, built from zero, with the maths of each and the experiments that tell you which one to pick.
The network on the right was pretrained in this page. Pick a method, set how many labels you have and how far the new task is from what the network already knows, and it trains live. Watch accuracy on familiar data, on shifted data, and on the old skills. Find where each method wins.
You need to know what a neural network layer and gradient descent are. The lesson Backbone and head is the gentle start. We build the rest from zero.
A real network trained in your browser: 16 fixed input sensors, two 16-wide tanh layers pretrained on three shape tasks, and a new head. Each run is the mean of three seeds, 300 steps of SGD with momentum. Training points come only from the left of the plane; the shaded right side is the shifted test set. All numbers are this toy's own, illustrative of the trends in the papers cited below, not their values.
Chapter 0
Meet the decision every transfer project faces, and see on a real-sized network what each option trains and stores
You work at a hospital. You have a vision model pretrained on 14 million everyday photos. You have 400 labelled chest X-rays. You want the model to flag one disease.
The model has about 86 million numbers in it, its weights. The pretraining run set every one of them. Now you must decide which of them may change on your 400 X-rays. That decision is the whole topic of this lesson.
Change too little, and the model cannot learn what is new about X-rays. Change too much, and 400 examples are not enough to set 86 million numbers well. The model memorizes your 400 images and may lose what it knew. Between those two failures lies a whole family of methods.
Here they are, from least to most change. We will build each one from zero.
A 12-layer transformer encoder the size of ViT-B or BERT-base: width d = 768, about 85 million weights. Pick a method. Warm parts train; purple parts stay frozen.
Counts are computed exactly for a 12-layer encoder with d = 768, an MLP of width 3,072 and a 10-class head (embeddings not counted). Adapters use bottleneck m = 64, two per layer, as in Houlsby et al.; LoRA uses rank 8 on the query and value matrices; prefix tuning uses 10 prefix vectors per layer for keys and values.
This lesson is part of The Builder's Toolkit, a series on reusable blocks. Every block gets the same five questions:
Chapter 1
Train only a head, then unfreeze the top k layers one by one, and count the price in weights, memory and time
Start with the least change possible. Keep every pretrained weight fixed. Run each input through the frozen network and take its last hidden vector, the feature. Then train one new linear layer on top that maps the feature to your classes. That new layer is the head. Training only the head is called a linear probe.
The data flow, for a batch of B images: pixels (B × 3 × 224 × 224) go into the frozen encoder. Out comes one feature per image (B × 768). The head is a 768 × 10 matrix plus 10 biases, and gives 10 scores per image (B × 10). Only those 7,690 numbers learn.
A probe cannot build new features. It can only read what the frozen network already computes. If the answer is there, a probe finds it with very little data. If it is not there, no amount of data helps the probe. The lesson Probing representations goes deep on reading a network this way.
The next step is to let the top k layers train as well. Why the top? Because the bottom layers of a pretrained network compute general things, like edges and textures in images or word pieces in text. The top layers are the most specific to the pretraining task. So they are the first to need a change.
Each transformer layer has an attention block and an MLP block. Count its weights with width d:
Worked numbers: d = 768, C = 10. One layer is 12 × 589,824 + 13 × 768 = 7,077,888 + 9,984 = 7,087,872. With k = 2: 2 × 7,087,872 + 7,690 = 14,183,434 numbers train. That is about 17% of the 85 million in the 12 layers.
Slide k, the number of top layers that train. Watch three costs: weights that train, optimizer memory, and the time of one training step.
Memory counts 12 bytes per trainable weight for its fp32 gradient and two Adam moments, on top of the weights themselves; activations are not counted. Step time assumes a backward pass costs about twice a forward pass and stops at the lowest trainable layer. Both are standard rules of thumb, not measurements.
Note the time bar. If nothing below layer 12 − k trains, backpropagation can stop there. The bottom layers then run forward only. So unfreezing the top two layers is cheap in time as well as memory. Methods that place small trainable parts in every layer, like adapters and LoRA, must send gradients down through every layer, even though few weights change.
Chapter 2
Insert small trainable modules, learned prefix vectors or bias-only updates, and see how each one changes the network without touching its weights
Last-k unfreezing changes whole layers. A different idea: keep every original weight frozen, and add a few new trainable numbers in clever places. The original network stays intact. Each new task only needs its own small set of new numbers. This family is called parameter-efficient fine-tuning, or PEFT.
Houlsby and colleagues proposed the adapter in 2019 Houlsby 2019 · §2.1. An adapter is a tiny two-layer network inserted inside each transformer layer. It squeezes the d-wide vector down to m numbers, applies a nonlinearity, and expands back to d. Then it adds the result to its own input, through a skip connection:
The paper counts 2md + d + m new numbers per adapter, and inserts two adapters per layer: one after attention, one after the MLP Houlsby 2019 · §2.1. Worked numbers: d = 768, m = 64. One adapter is 2 × 64 × 768 + 768 + 64 = 99,136. Two per layer, 12 layers: 2,379,264, about 2.8% of the encoder.
The skip connection is the key design choice. If Wup starts near zero, the adapter adds almost nothing, so h′ ≈ h. The network starts exactly as pretrained, and training moves it away gently. The paper calls this a near-identity initialization Houlsby 2019 · §2.1.
Forty 16-number vectors pass through an adapter; the bars show one of them. The task needs a change that uses three directions. Set the bottleneck m and how large Wup starts, then press Train.
A toy with d = 16 trained in this page by gradient descent on squared error. The target change is a fixed rank-3 map, chosen to make the bottleneck limit visible.
Adapters have one cost that never goes away. They are extra layers in the forward pass. At inference you run them for every token. The LoRA paper measured GPT-2 medium at batch size 1 and 128 tokens: 19.8 ms without adapters, 23.9 ms and 25.8 ms with two adapter designs, which is 20.7% and 30.3% slower Hu 2021 · Table 1. With large batches and long sequences the slow-down shrank to 2 to 3% Hu 2021 · Table 1.
A second idea leaves even the layers alone. In a transformer, every token attends to every other token. So add a few extra "tokens" whose vectors you learn. The frozen network attends to them as if they were part of the text, and they steer its behaviour.
Prompt tuning adds L learned vectors to the input sequence only Lester 2021 · §2. Prefix tuning adds learned key and value vectors at every layer Li & Liang 2021 · §4. A prefix only changes where attention goes, through the softmax:
Four frozen tokens and one query. Drag the warm prefix key on the plane, and set its value with the slider. Try to move the output onto the target line.
A one-head toy in two dimensions with scalar values, computed exactly. Real prefixes are d-wide vectors learned by gradient descent, at every layer.
The parameter counts are tiny. Ten prefix vectors per layer for keys and values, d = 768, 12 layers: 2 × 10 × 768 × 12 = 184,320. Lester et al. tuned a prompt of 20,480 numbers per task on an 11-billion-parameter T5 model, and found prompt tuning closes the gap to full fine-tuning as models grow to billions of parameters Lester 2021 · §1, §3.1. Prefix tuning, with about 0.1% of the parameters, matched fine-tuning on full data and beat it in low-data settings on its generation tasks Li & Liang 2021 · abstract.
The simplest PEFT method of all: freeze every weight matrix, and train only the bias vectors, the numbers added after each matrix multiply. On BERT-large these are 0.08% of the parameters. BitFit scored 80.9 on the GLUE test set, against 81.8 for full fine-tuning and 81.1 for adapters Ben-Zaken 2022 · Table 1. Its authors read this as evidence that fine-tuning mostly exposes knowledge the pretrained model already has Ben-Zaken 2022 · abstract.
Chapter 3
Write the change to a weight matrix as the product of two thin matrices, count what that saves, and see how much of a change a given rank can hold
Fine-tuning a weight matrix W0 means learning a change ΔW, so the new matrix is W0 + ΔW. Full fine-tuning lets ΔW be anything: for a 768 × 768 matrix, 589,824 free numbers.
Hu and colleagues asked a sharp question in 2021. Does ΔW really need all that freedom? Their bet: the change needed for a new task has a low "intrinsic rank" Hu 2021 · §4.1. So they forced it to be low rank, by writing it as a product of two thin matrices. They called it LoRA, low-rank adaptation.
The rank of a matrix is the number of independent directions it can write into. A rank-1 matrix is a column times a row, b aT. Feed it any input x: the output is always the column b, scaled by the number a·x. So a rank-1 change can only push outputs along one direction. A rank-r matrix is a sum of r such pieces, so it can push along r directions.
Count: B has d·r numbers and A has r·k, so LoRA trains r(d + k) numbers instead of d·k. Worked numbers: a 768 × 768 attention matrix with r = 8. LoRA trains 8 × (768 + 768) = 12,288 numbers instead of 589,824, which is 2.1%. Put rank 8 on the query and value matrices of all 12 layers: 2 × 12 × 12,288 = 294,912.
The data flow inside one LoRA layer, for a batch of B sequences of T tokens: the input x is B × T × 768. The frozen path gives W0x, B × T × 768. The side path first computes Ax, only B × T × 8, then B(Ax), back to B × T × 768. The two outputs are added. The side path never builds the full 768 × 768 matrix.
The left grid is a "true" 16 × 16 weight change. The middle grid is the best rank-r version of it; the right grid is what is left over. Slide r, and slide how spread out the change is.
The true change is built as U·diag(σ)·VT with random orthogonal U and V and σi = e−i/τ, so its singular values are known exactly. By the Eckart–Young theorem, keeping the top r of them is the best rank-r approximation.
The error you see follows one rule. Write the change by its singular values σ1 ≥ σ2 ≥ …, the strengths of its independent directions. The best rank-r version keeps the top r and drops the rest. The share it misses is:
Worked numbers: singular values 1, 0.5, 0.25, 0.125 and nothing else. Rank 2 misses 0.252 + 0.1252 = 0.078 of a total 1 + 0.25 + 0.0625 + 0.0156 = 1.328. err = √(0.078 / 1.328) = 0.24, so rank 2 holds most of the change. If the singular values decay fast, a small rank is enough. Chapter 5 asks whether real fine-tuning changes decay fast.
Chapter 4
Train a LoRA patch live and see why B starts at zero, what the α/r scale is for, and how merging removes all inference cost
The full LoRA forward pass has one more piece, a fixed scale:
All three details are in the paper's method section: random Gaussian A, zero B, so ΔW = BA is zero at the start, and the update scaled by α/r Hu 2021 · §4.1. The authors set α to the first r they tried and did not tune it, because with Adam, tuning α is roughly like tuning the learning rate Hu 2021 · §4.1.
Why zero B? At step 0 the network computes exactly what the pretrained network computes. Training starts from what the model knows, not from a random perturbation of it. Why not zero A too? Because then both gradients would be zero forever. The gradient of B is proportional to Ax, and the gradient of A is proportional to B. With B = 0 and A random, B moves first, and then A starts moving.
A frozen 16 × 16 layer must learn a rank-3 change. Four LoRA patches train at once, ranks 1, 2, 4 and 8, with Adam. Switch the two details off, one at a time, and press Train.
Computed in this page: 64 random inputs, mean squared error to the target outputs, Adam with learning rate 0.01, α = 4, 200 steps. The dashed line is the loss of the frozen layer with no patch, the "pretrained model".
Three things to see. With B at zero, every curve starts on the dashed line: step 0 is the pretrained model. With B random, the curves start above it: the patch first damages the model, then repairs it. With the α/r scale on, the four ranks start down the curve together. Schulman and Thinking Machines report the same thing for real LLMs: with the 1/r factor, "the learning curve is exactly the same at the beginning of training, regardless of rank" LoRA Without Regret · 2025. Rank 1 and 2 then stall above zero, because the target change has rank 3.
After training, compute W = W0 + (α/r)BA once, and store it. The model is now an ordinary network with the same shapes as before. The paper stresses this: LoRA adds no inference latency "by construction" Hu 2021 · §4.1. To switch tasks, subtract BA and add another task's B′A′. Press Merge above to check that the merged matrix gives the same outputs.
The savings are large at scale. On GPT-3 175B, LoRA cut training memory from 1.2 TB to 350 GB, and with r = 4 on the query and value matrices cut each checkpoint from 350 GB to 35 MB Hu 2021 · §4.2. Training ran 25% faster than full fine-tuning, because most gradients are never computed Hu 2021 · §4.2.
Chapter 5
Train the whole network inside a random subspace of d directions, find the smallest d that works, and read the evidence from real language models
LoRA rests on a bet: a good fine-tune lives in a small space. Can we measure how small? Aghajanyan, Zettlemoyer and Gupta did, in 2020 Aghajanyan 2020 · §3.
The trick is to forbid the model from moving freely. Pick d random directions in the space of all D weights. Let training move the weights only along those directions:
The intrinsic dimension d90 is the smallest d that reaches 90% of the score full fine-tuning gets Aghajanyan 2020 · §3. Worked example from the paper's own definition: if full fine-tuning reaches 85% accuracy, d90 is the smallest d that reaches 0.9 × 85% = 76.5% Aghajanyan 2020 · §3.
The hero's network has 561 weights. The task needs real adaptation (domain gap 0.6, 256 labels). Press Run: it fine-tunes inside random subspaces of 1, 2, 4, … directions. Then slide to read each result.
Computed in this page: 250 Adam steps on z for each d, one seed, P Gaussian with variance 1/D. The head is part of θ and starts at zero. Your browser's numbers are the toy's own, not the paper's.
The real numbers are striking. On the MRPC paraphrase task, RoBERTa-Large reached 90% of full fine-tuning with about 200 trainable numbers out of some 355 million Aghajanyan 2020 · abstract. Their Table 1 lists d90 = 207 for MRPC and 774 for QQP with RoBERTa-Large, and 1,608 for MRPC with the smaller BERT-Base Aghajanyan 2020 · Table 1. They also found that pretraining lowers intrinsic dimension, and that larger models tend to have lower intrinsic dimension Aghajanyan 2020 · abstract.
That is the "why" behind LoRA. If a few hundred directions in the whole weight space are enough, then a rank-8 change per matrix is generous. The LoRA paper saw this directly: on GPT-3 175B, adapting the query and value matrices with r = 1 scored 73.4 on WikiSQL and 91.3 on MultiNLI, against 73.5 and 91.4 with r = 64 Hu 2021 · Table 6.
Chapter 6
See how a random head pushes the backbone in useless directions, why that hurts shifted data and old skills, and how probing first fixes it
Two things can go wrong when every weight is free to move. They have the same cause.
Catastrophic forgetting: the network gets better at the new task and worse at what it could do before. Howard and Ruder named the risk directly: "Overly aggressive fine-tuning will cause catastrophic forgetting" Howard & Ruder 2018 · §3.3.
Feature distortion: the network gets better on data like its training set, the in-distribution (ID) data, and worse on shifted data, the out-of-distribution (OOD) data. Kumar, Raghunathan, Jones, Ma and Liang showed this in 2022 Kumar 2022 · abstract.
They compared full fine-tuning (FT) and linear probing (LP) on 10 distribution shifts, from CIFAR-10 to STL, to ImageNet sketches and satellite images, using good pretrained features (MoCo-v2 and CLIP) Kumar 2022 · §4. Averaged over the shifts:
| Method | ID accuracy (avg of 6) | OOD accuracy (avg of 10) |
|---|---|---|
| Linear probe (LP) | 82.9% | 66.2% |
| Full fine-tune (FT) | 85.1% | 59.3% |
| Probe, then fine-tune (LP-FT) | 85.7% | 68.9% |
From Kumar et al. 2022, Tables 1 and 2. Linear probing beat fine-tuning OOD on 8 of the 10 shifts; the two exceptions, CIFAR-10.1 and ImageNetV2, are shifts built to closely copy their ID data Kumar 2022 · §4.1.
Fine-tuning won ID by about 2 points and lost OOD by about 7. On ImageNet-R (renditions such as art and cartoons), FT got 52.4% against LP's 70.6% Kumar 2022 · Table 2. And this was not plain overfitting. Even when they picked the fine-tuning checkpoint by OOD accuracy, which you cannot do in practice, LP still won, 67.1% against 61.3% Kumar 2022 · §4.3.
Look at the gradient that reaches the backbone. The head scores a feature z with a weight vector v: score = v·z. With a logistic loss, the gradient with respect to the feature is:
Worked numbers: at step 0 the random head is unsure, p ≈ 0.5. For a class-1 example, p − y = −0.5. So every feature is pushed along −0.5·v, a random direction chosen by a random head. The backbone bends its features to suit a head that knows nothing yet. It bends them only for the ID training inputs. Shifted inputs, which training never sees, are left behind with features that no longer match the head Kumar 2022 · §1.
The fix follows from the formula. Train the head first, with the backbone frozen: a linear probe. Now p ≈ y on most examples, so (p − y) is small, and the backbone gets small, meaningful gradients. Then fine-tune everything. This is LP-FT. In the paper, LP-FT changed features about 20 times less than fine-tuning did, on both ID and OOD inputs Kumar 2022 · §4.3.
The hero's network, 48 labels. Three methods train side by side. Raise the size of the random starting head and watch fine-tuning's features drift. Raise the domain gap to see when fine-tuning's extra freedom is needed.
Each bar is the mean of three seeds. "Old skills kept" runs the three original pretraining heads on the new features. "Feature drift" is the mean distance between each test point's feature before and after training, the same measurement Kumar et al. use. Illustrative toy numbers.
Chapter 7
Give lower layers smaller learning rates, unfreeze from the top down, and see the trade between protecting old features and learning new ones
Freezing is all or nothing: a weight either moves or it does not. There is a softer dial. Let every layer train, but at a different speed.
Howard and Ruder introduced this in ULMFiT in 2018, under the name discriminative fine-tuning Howard & Ruder 2018 · §3.2. Each layer l gets its own learning rate ηl. Their recipe: choose the rate for the last layer by fine-tuning the last layer alone, then divide by 2.6 for each layer below:
Worked numbers for a 12-layer model with η12 = 10−4: layer 11 gets 3.8 × 10−5, layer 6 gets 10−4 / 2.66 = 3.2 × 10−7, and layer 1 gets 10−4 / 2.611 = 2.7 × 10−9. The bottom layers barely move. That is the point: they hold the most general features.
Full fine-tuning of the hero's network, 48 labels. Layer 2 trains at rate/ξ and layer 1 at rate/ξ2. ξ = 1 is plain fine-tuning. Move ξ, then move the domain gap.
Mean of three seeds, 300 steps of SGD with momentum. Ghost marks show plain fine-tuning (ξ = 1). Illustrative toy numbers.
ULMFiT pairs this with gradual unfreezing: first train only the last layer for one epoch, then unfreeze the next layer down and train again, and so on, until all layers train Howard & Ruder 2018 · §3.3. Both ideas say the same thing as LP-FT: settle the top before you disturb the bottom. With these tricks, ULMFiT with only 100 labelled examples matched training from scratch on 100 times more data Howard & Ruder 2018 · abstract.
Chapter 8
Freeze with requires_grad_, write a LoRA layer from scratch in about 20 lines, inject it into a ViT, then do the same with one peft call
Everything in this lesson comes down to one flag per weight. In PyTorch every parameter has requires_grad. If it is False, autograd computes no gradient for it, and the optimizer leaves it alone. The rest is bookkeeping. The lesson PyTorch autograd explains the flag.
import torch
from transformers import ViTForImageClassification
model = ViTForImageClassification.from_pretrained(
"google/vit-base-patch16-224-in21k", num_labels=10) # 12 layers, d = 768
# Linear probe: freeze everything, then let only the new head train
model.requires_grad_(False)
model.classifier.requires_grad_(True) # Linear(768 -> 10): 7,690 numbers
# Last-k: also unfreeze the top k blocks and the final norm
k = 2
for block in model.vit.encoder.layer[-k:]:
block.requires_grad_(True) # 7,087,872 numbers each
model.vit.layernorm.requires_grad_(True)
n = sum(p.numel() for p in model.parameters() if p.requires_grad)
print(f"trainable: {n:,}")
# Discriminative rates (ULMFiT): each block down gets rate / 2.6
model.requires_grad_(True)
top_lr, xi = 1e-4, 2.6
head = list(model.classifier.parameters()) + list(model.vit.layernorm.parameters())
groups = [{"params": head, "lr": top_lr}]
for depth, block in enumerate(reversed(model.vit.encoder.layer), start=1):
groups.append({"params": block.parameters(), "lr": top_lr / xi**depth})
groups.append({"params": model.vit.embeddings.parameters(), "lr": top_lr / xi**13})
opt = torch.optim.AdamW(groups, weight_decay=0.01)
A LoRA layer wraps a frozen nn.Linear and adds the side path. Read the shapes in the comments: the side path goes from din down to r and back up to dout.
import math
import torch
import torch.nn as nn
class LoRALinear(nn.Module):
"""y = W0 x + b + (alpha / r) * B A x, with W0 and b frozen."""
def __init__(self, base: nn.Linear, r=8, alpha=16):
super().__init__()
self.base = base.requires_grad_(False) # W0: (d_out, d_in)
d_in, d_out = base.in_features, base.out_features
self.A = nn.Parameter(torch.empty(r, d_in)) # (r, d_in), random
self.B = nn.Parameter(torch.zeros(d_out, r)) # (d_out, r), zero: BA = 0
nn.init.kaiming_uniform_(self.A, a=math.sqrt(5))
self.scale = alpha / r
self.merged = False
def forward(self, x): # x: (B, T, d_in)
y = self.base(x) # (B, T, d_out)
if not self.merged:
side = (x @ self.A.T) @ self.B.T # (B,T,r) -> (B,T,d_out)
y = y + self.scale * side
return y
@torch.no_grad()
def merge(self): # fold the patch into W0
self.base.weight += self.scale * (self.B @ self.A)
self.merged = True
Now swap it in. The attention block of a Hugging Face ViT names its projections query, key and value. We replace two of them in every layer:
def add_lora(model, names=("query", "value"), r=8, alpha=16):
model.requires_grad_(False) # freeze everything first
for module in list(model.modules()):
for child_name, child in list(module.named_children()):
if child_name in names and isinstance(child, nn.Linear):
setattr(module, child_name, LoRALinear(child, r, alpha))
return model
model = add_lora(model) # 12 x 2 x 12,288 = 294,912
model.classifier.requires_grad_(True) # the new head always trains
opt = torch.optim.AdamW(
[p for p in model.parameters() if p.requires_grad], lr=1e-3)
# after training: merge, and the model is a plain ViT again
for m in model.modules():
if isinstance(m, LoRALinear):
m.merge()
The peft library does the same injection, plus saving and loading only the small patch:
from peft import LoraConfig, get_peft_model
config = LoraConfig(r=8, lora_alpha=16, lora_dropout=0.1,
target_modules=["query", "value"], # module names to patch
modules_to_save=["classifier"]) # the head trains in full
model = get_peft_model(model, config)
model.print_trainable_parameters() # about 0.35% of the model
# ... train exactly as before ...
model.save_pretrained("xray-lora") # a few MB, not 340 MB
model = model.merge_and_unload() # W = W0 + (alpha/r) B A
Which modules should you patch? The original paper patched only attention weights, and found query plus value best under a fixed budget Hu 2021 · Table 5. Later work changed the default. Thinking Machines found that "attention-only LoRA significantly underperforms MLP-only LoRA", and that adding attention on top of the MLP gave no further benefit LoRA Without Regret · 2025. For language models, patch every linear layer, MLP included. Use the device to see what that costs.
Pick a model, which matrices to patch, and the rank. The count is exact: r × (din + dout) per patched matrix.
The 8B decoder uses a Llama-3-8B-style shape: d = 4,096, MLP width 14,336 with gate, up and down projections, key and value width 1,024 (grouped-query attention), 32 layers; its total includes embeddings. File size assumes 2 bytes per number.
Chapter 9
Compare against a linear probe, track shifted data, measure which layers changed, read the learning curves, and run the shuffled-label test
A fine-tune can look fine on one number and be broken underneath. Five checks catch most problems. Run them every time.
Worked numbers: a 768 × 768 matrix with entries of typical size 0.02 has &lVert;W0&rVert;F ≈ 0.02 × 768 = 15.4. If fine-tuning moved each entry by about 0.001, &lVert;ΔW&rVert;F ≈ 0.001 × 768 = 0.77, so δ ≈ 5%.
Domain gap 0.5, 96 labels. Pick a method and press Train. Then switch on Shuffle labels and train again.
One seed, 400 steps, accuracy measured every 10 steps on 500 held-out familiar points and 400 shifted points. This tiny toy trains with a large learning rate, so its weights move far more than in a real fine-tune, where a few percent is typical. Illustrative toy numbers.
import copy, torch ref = copy.deepcopy(model.state_dict()) # snapshot before training # ... fine-tune ... @torch.no_grad() def weight_change(model, ref): """Relative Frobenius change of every 2-D weight (merge LoRA first).""" out = {} for name, w in model.named_parameters(): if w.ndim == 2 and name in ref: w0 = ref[name].to(w.device) out[name] = ((w - w0).norm() / w0.norm()).item() return out for name, d in sorted(weight_change(model, ref).items()): print(f"{d:7.4f} {name}") # bottom layers should move least
Chapter 10
Follow the same decision through audio, medical imaging, language models, robot policies and satellite images, then read the decision chart
The question "how much to unfreeze" has the same three inputs in every field: how many labels you have, how far the new data is from the pretraining data, and how much compute and latency you can spend. Here is the same block in five fields.
| Field | Pretrained model | Typical choice | Why |
|---|---|---|---|
| Audio | CLAP or an audio foundation model | Frozen encoder + small head; LoRA if the sounds are unusual | Embeddings already separate many sound events; labels for a rare sound are few. See CLAP and the frozen-probe chapter of AudioFM. |
| Medical images | ImageNet-trained CNN or ViT | Fine-tune deeper, or LoRA at higher rank; probe as baseline | A big domain gap. Raghu et al. found that on two large medical tasks, ImageNet transfer offered little gain in final accuracy, and small models did as well Raghu 2019 · abstract. |
| LLM instruction tuning | A base language model | LoRA on every linear layer; QLoRA if memory is tight | Small-to-medium datasets; many tasks share one base. QLoRA fine-tuned a 65B model on one 48 GB GPU Dettmers 2023 · abstract. |
| Robot policies | A vision-language-action model | LoRA on all linear layers | OpenVLA: LoRA r = 32 reached 68.2% success against 69.7% for full fine-tuning, training 1.4% of the parameters; last-layer-only reached 30.3% Kim 2024 · Table 1. |
| Satellite images | A remote-sensing or ImageNet model | LP-FT; watch region and time shift | On FMoW, test regions differ from training. Kumar et al.: OOD 32.0% for FT, 36.6% for LP, 36.8% for LP-FT Kumar 2022 · Table 2. |
The robot result is worth a second look. OpenVLA's authors found that freezing the vision encoder gave 47.0% success, and that "further adaptation of the visual features to the target scene is crucial" Kim 2024 · §5.3. LoRA on all layers did that cheaply, at 59.7 GB of GPU memory against 163.3 GB for full fine-tuning Kim 2024 · Table 1. The lesson microVLA builds these models.
Each region names the method this lesson recommends. Tap the chart, or pick a field. Then add a constraint and see the chart change.
A judgement chart that summarizes this lesson's evidence; the borders are rules of thumb, not measured lines. Field positions are typical cases, not fixed facts.
| Avoid | When | Use instead |
|---|---|---|
| Full fine-tuning | Tiny data on a huge model; you care about shifted data or old skills | A probe, then LoRA or LP-FT. A random head on full fine-tuning distorts features (Chapter 6). |
| Low-rank LoRA | A big domain shift, or a dataset larger than the patch can hold | Higher rank on every layer, or full fine-tuning. Thinking Machines: when datasets exceed LoRA's capacity, "LoRA underperforms FullFT" LoRA Without Regret · 2025. See the LoRA Without Regret teardown. |
| LoRA at very large batch | Training with very large batches | Smaller batches, or full fine-tuning. LoRA "is less tolerant of large batch sizes than FullFT", and higher rank does not fix it LoRA Without Regret · 2025. |
| Adapters | Latency matters at batch size 1 | LoRA, merged into the weights: +0 ms against +20.7% to +30.3% for adapters in the LoRA paper's test Hu 2021 · Table 1. |
| A linear probe alone | The answer is not in the frozen features (big gap) | Unfreeze something. In the hero, the probe stalls near 70% at a large gap however many labels you add. |
Chapter 11
The memory bill for each method, a one-screen cheat sheet, the honest limits, and where to go next
One last cost decides many real projects: GPU memory. Full fine-tuning with Adam holds, for every weight, the weight, its gradient and two optimizer moments. PEFT methods hold optimizer state only for the small patch. The LoRA paper's GPT-3 numbers show the scale: 1.2 TB down to 350 GB Hu 2021 · §4.2.
Worked numbers for an 8-billion-parameter model: full fine-tuning needs about 16 × 8 × 109 = 128 GB before activations. LoRA with 42 million trained numbers needs 2 × 8 × 109 + 16 × 42 × 106 ≈ 16.7 GB. With a 4-bit base, about 4.7 GB.
Slide the model size. Each bar is the memory for weights and optimizer state, against common GPU sizes.
The formula above; LoRA trains 0.5% of the weights. Activations, the KV cache and framework overhead are not counted, so real runs need more. A rule of thumb, not a measurement.
| Method | What trains (d = 768, 12 layers) | Inference cost | Best when |
|---|---|---|---|
| Linear probe | Head only: dC + C (7,690) | None extra | Few labels, small gap; always the baseline |
| Last k layers | k(12d2 + 13d) + head (14.2M for k = 2) | None extra | Medium data, gap in high-level features |
| BitFit | Biases only (about 0.1%) | None extra | Small to medium data, smallest storage |
| Prompt / prefix | L·d at input, or 2·L·d per layer (184,320 for L = 10) | Longer sequence | Very large models, many tasks |
| Adapters | 2(2md + d + m) per layer (2.4M for m = 64) | Extra layers: slower at batch 1 | Many tasks, latency not critical |
| LoRA | r(din + dout) per matrix (294,912 for r = 8 on q, v) | None after merging | Most cases; patch every linear layer |
| LP-FT | Head, then everything | None extra | Enough data to fine-tune, and shifted data matters |
| Full fine-tune | Everything (85M) | None extra, but one full copy per task | Large data, large gap, memory to spare |
The block in one line: the maths is W = W0 + (α/r)BA with B = 0 at the start; the knobs are what to unfreeze, rank r, α, learning rate (about 10× full fine-tuning for LoRA) and per-layer decay; the failures are forgetting, feature distortion from a random head, too low a rank for a big gap, and adapter latency; the alternatives are a probe, LP-FT, and full fine-tuning.
The toy in this page has 561 weights; real models have billions. Its trends match the papers, but its numbers are its own. The LP-FT theory is proved for two-layer linear networks Kumar 2022 · §3; the deep-network evidence is empirical. Intrinsic dimension was measured with random projections, so the true value is likely lower than reported Aghajanyan 2020 · §4.2. And the decision chart is judgement, not law: when in doubt, run a probe and a LoRA and compare.
Now press Present or Teach and explain, from memory, why fine-tuning every weight can lose to a linear probe on shifted data, and what LoRA's zero-initialized B buys you. Then go back to the hero and find the domain gap where the probe stops being enough.