How Much to Unfreeze

You have a pretrained network and a new task. You can train only a new head, a few top layers, a small low-rank patch, or every weight. Each choice changes what you gain on new data, what you lose on shifted data, and what the model forgets.

Probes, adapters, LoRA, prefix and prompt tuning, BitFit and full fine-tuning, built from zero, with the maths of each and the experiments that tell you which one to pick.

The network on the right was pretrained in this page. Pick a method, set how many labels you have and how far the new task is from what the network already knows, and it trains live. Watch accuracy on familiar data, on shifted data, and on the old skills. Find where each method wins.

You need to know what a neural network layer and gradient descent are. The lesson Backbone and head is the gentle start. We build the rest from zero.

Unfreeze, and watch what happens

Ready
32
none
4

A real network trained in your browser: 16 fixed input sensors, two 16-wide tanh layers pretrained on three shape tasks, and a new head. Each run is the mean of three seeds, 300 steps of SGD with momentum. Training points come only from the left of the plane; the shaded right side is the shifted test set. All numbers are this toy's own, illustrative of the trends in the papers cited below, not their values.

Chapter 0

One model, seven ways to adapt it

Meet the decision every transfer project faces, and see on a real-sized network what each option trains and stores

You work at a hospital. You have a vision model pretrained on 14 million everyday photos. You have 400 labelled chest X-rays. You want the model to flag one disease.

The model has about 86 million numbers in it, its weights. The pretraining run set every one of them. Now you must decide which of them may change on your 400 X-rays. That decision is the whole topic of this lesson.

Change too little, and the model cannot learn what is new about X-rays. Change too much, and 400 examples are not enough to set 86 million numbers well. The model memorizes your 400 images and may lose what it knew. Between those two failures lies a whole family of methods.

Here they are, from least to most change. We will build each one from zero.

What each method trains

A 12-layer transformer encoder the size of ViT-B or BERT-base: width d = 768, about 85 million weights. Pick a method. Warm parts train; purple parts stay frozen.

Counts are computed exactly for a 12-layer encoder with d = 768, an MLP of width 3,072 and a 10-class head (embeddings not counted). Adapters use bottleneck m = 64, two per layer, as in Houlsby et al.; LoRA uses rank 8 on the query and value matrices; prefix tuning uses 10 prefix vectors per layer for keys and values.

The five questions this lesson answers

This lesson is part of The Builder's Toolkit, a series on reusable blocks. Every block gets the same five questions:

Why it worksThe maths: low rank, intrinsic dimension, feature distortion.
Build itShapes in and out, LoRA from scratch in PyTorch, then the one-liner.
Where elseAudio, medical images, LLMs, robots, satellites.
When notTiny data on a huge model, too low a rank, latency.
Did it work?Probe baseline, shifted-data accuracy, weight changes, curves.
  1. Probes and the last few layers: the cheapest options, and what they cost in memory and time.
  2. Add a little: adapters, prompts and biases: train small new parts and leave the old ones alone.
  3. LoRA: a low-rank patch: why a change to a weight matrix can be stored as two thin matrices.
  4. LoRA's small print: zero init, the α/r scale, and merging for free inference.
  5. Why low rank is enough: intrinsic dimension, measured in this page.
  6. Forgetting and distorted features: why full fine-tuning can lose on shifted data, and the LP-FT fix.
  7. Different speeds for different layers: discriminative learning rates and gradual unfreezing.
  8. Build it in PyTorch: freezing, LoRA from scratch, then peft.
  9. Did it work?: the checks that catch a bad fine-tune.
  10. Where else it works, and when not to use it: five fields and a decision chart.
  11. Connections: the cheat sheet, the memory bill, and where to go next.
You have 400 labelled X-rays and an 86-million-weight model. Why is "train every weight" not automatically the best choice?

Chapter 1

Probes and the last few layers

Train only a head, then unfreeze the top k layers one by one, and count the price in weights, memory and time

Start with the least change possible. Keep every pretrained weight fixed. Run each input through the frozen network and take its last hidden vector, the feature. Then train one new linear layer on top that maps the feature to your classes. That new layer is the head. Training only the head is called a linear probe.

The data flow, for a batch of B images: pixels (B × 3 × 224 × 224) go into the frozen encoder. Out comes one feature per image (B × 768). The head is a 768 × 10 matrix plus 10 biases, and gives 10 scores per image (B × 10). Only those 7,690 numbers learn.

A probe cannot build new features. It can only read what the frozen network already computes. If the answer is there, a probe finds it with very little data. If it is not there, no amount of data helps the probe. The lesson Probing representations goes deep on reading a network this way.

Unfreezing the top k layers

The next step is to let the top k layers train as well. Why the top? Because the bottom layers of a pretrained network compute general things, like edges and textures in images or word pieces in text. The top layers are the most specific to the pretraining task. So they are the first to need a change.

Each transformer layer has an attention block and an MLP block. Count its weights with width d:

P(k)
How many numbers train when the top k layers and the head are unfrozen.
k
How many of the top layers you unfreeze. k = 0 is a linear probe.
12d2 + 13d
One layer: four d × d attention matrices (4d2), an MLP from d to 4d and back (8d2), and their biases and norm parameters (13d).
dC + C
The new head for C classes.

Worked numbers: d = 768, C = 10. One layer is 12 × 589,824 + 13 × 768 = 7,077,888 + 9,984 = 7,087,872. With k = 2: 2 × 7,087,872 + 7,690 = 14,183,434 numbers train. That is about 17% of the 85 million in the 12 layers.

Unfreeze from the top

Slide k, the number of top layers that train. Watch three costs: weights that train, optimizer memory, and the time of one training step.

2

Memory counts 12 bytes per trainable weight for its fp32 gradient and two Adam moments, on top of the weights themselves; activations are not counted. Step time assumes a backward pass costs about twice a forward pass and stops at the lowest trainable layer. Both are standard rules of thumb, not measurements.

Note the time bar. If nothing below layer 12 − k trains, backpropagation can stop there. The bottom layers then run forward only. So unfreezing the top two layers is cheap in time as well as memory. Methods that place small trainable parts in every layer, like adapters and LoRA, must send gradients down through every layer, even though few weights change.

With d = 768, how many weights are in one transformer layer (attention + MLP + biases and norms)?

Chapter 2

Add a little: adapters, prompts and biases

Insert small trainable modules, learned prefix vectors or bias-only updates, and see how each one changes the network without touching its weights

Last-k unfreezing changes whole layers. A different idea: keep every original weight frozen, and add a few new trainable numbers in clever places. The original network stays intact. Each new task only needs its own small set of new numbers. This family is called parameter-efficient fine-tuning, or PEFT.

Adapters: a detour with a skip

Houlsby and colleagues proposed the adapter in 2019 Houlsby 2019 · §2.1. An adapter is a tiny two-layer network inserted inside each transformer layer. It squeezes the d-wide vector down to m numbers, applies a nonlinearity, and expands back to d. Then it adds the result to its own input, through a skip connection:

h
The layer's output vector, d numbers. Frozen layers made it.
Wdown
An m × d matrix (plus m biases) that squeezes h to m numbers. m is the bottleneck size, much smaller than d.
Wup
A d × m matrix (plus d biases) that expands back to d. f is a nonlinearity such as ReLU.
h′
The adapted vector, passed on to the next layer.

The paper counts 2md + d + m new numbers per adapter, and inserts two adapters per layer: one after attention, one after the MLP Houlsby 2019 · §2.1. Worked numbers: d = 768, m = 64. One adapter is 2 × 64 × 768 + 768 + 64 = 99,136. Two per layer, 12 layers: 2,379,264, about 2.8% of the encoder.

The skip connection is the key design choice. If Wup starts near zero, the adapter adds almost nothing, so h′ ≈ h. The network starts exactly as pretrained, and training moves it away gently. The paper calls this a near-identity initialization Houlsby 2019 · §2.1.

Train an adapter

Forty 16-number vectors pass through an adapter; the bars show one of them. The task needs a change that uses three directions. Set the bottleneck m and how large Wup starts, then press Train.

2
0.00

A toy with d = 16 trained in this page by gradient descent on squared error. The target change is a fixed rank-3 map, chosen to make the bottleneck limit visible.

Adapters have one cost that never goes away. They are extra layers in the forward pass. At inference you run them for every token. The LoRA paper measured GPT-2 medium at batch size 1 and 128 tokens: 19.8 ms without adapters, 23.9 ms and 25.8 ms with two adapter designs, which is 20.7% and 30.3% slower Hu 2021 · Table 1. With large batches and long sequences the slow-down shrank to 2 to 3% Hu 2021 · Table 1.

Prefix and prompt tuning: learn the input, not the weights

A second idea leaves even the layers alone. In a transformer, every token attends to every other token. So add a few extra "tokens" whose vectors you learn. The frozen network attends to them as if they were part of the text, and they steer its behaviour.

Prompt tuning adds L learned vectors to the input sequence only Lester 2021 · §2. Prefix tuning adds learned key and value vectors at every layer Li & Liang 2021 · §4. A prefix only changes where attention goes, through the softmax:

softmax(q·ki)
How much the query q attends to token i. The weights over all tokens add up to 1.
vi
The value token i passes on when attended to.
prefix
The learned key and value vectors. They compete for attention with the real tokens.
y
The attention output for this query.

Steer attention with one prefix

Four frozen tokens and one query. Drag the warm prefix key on the plane, and set its value with the slider. Try to move the output onto the target line.

1.60

A one-head toy in two dimensions with scalar values, computed exactly. Real prefixes are d-wide vectors learned by gradient descent, at every layer.

The parameter counts are tiny. Ten prefix vectors per layer for keys and values, d = 768, 12 layers: 2 × 10 × 768 × 12 = 184,320. Lester et al. tuned a prompt of 20,480 numbers per task on an 11-billion-parameter T5 model, and found prompt tuning closes the gap to full fine-tuning as models grow to billions of parameters Lester 2021 · §1, §3.1. Prefix tuning, with about 0.1% of the parameters, matched fine-tuning on full data and beat it in low-data settings on its generation tasks Li & Liang 2021 · abstract.

BitFit: train only the biases

The simplest PEFT method of all: freeze every weight matrix, and train only the bias vectors, the numbers added after each matrix multiply. On BERT-large these are 0.08% of the parameters. BitFit scored 80.9 on the GLUE test set, against 81.8 for full fine-tuning and 81.1 for adapters Ben-Zaken 2022 · Table 1. Its authors read this as evidence that fine-tuning mostly exposes knowledge the pretrained model already has Ben-Zaken 2022 · abstract.

Why does an adapter with Wup initialized near zero start training from exactly the pretrained model?

Chapter 3

LoRA: a low-rank patch on a frozen matrix

Write the change to a weight matrix as the product of two thin matrices, count what that saves, and see how much of a change a given rank can hold

Fine-tuning a weight matrix W0 means learning a change ΔW, so the new matrix is W0 + ΔW. Full fine-tuning lets ΔW be anything: for a 768 × 768 matrix, 589,824 free numbers.

Hu and colleagues asked a sharp question in 2021. Does ΔW really need all that freedom? Their bet: the change needed for a new task has a low "intrinsic rank" Hu 2021 · §4.1. So they forced it to be low rank, by writing it as a product of two thin matrices. They called it LoRA, low-rank adaptation.

Rank, in one picture

The rank of a matrix is the number of independent directions it can write into. A rank-1 matrix is a column times a row, b aT. Feed it any input x: the output is always the column b, scaled by the number a·x. So a rank-1 change can only push outputs along one direction. A rank-r matrix is a sum of r such pieces, so it can push along r directions.

ΔW
The change added to the frozen d × k matrix W0.
A
The "down" matrix: it reads the k-wide input into r numbers.
B
The "up" matrix: it writes those r numbers back out along r directions in the d-wide output.
r
The rank: how many directions the change can use.

Count: B has d·r numbers and A has r·k, so LoRA trains r(d + k) numbers instead of d·k. Worked numbers: a 768 × 768 attention matrix with r = 8. LoRA trains 8 × (768 + 768) = 12,288 numbers instead of 589,824, which is 2.1%. Put rank 8 on the query and value matrices of all 12 layers: 2 × 12 × 12,288 = 294,912.

The data flow inside one LoRA layer, for a batch of B sequences of T tokens: the input x is B × T × 768. The frozen path gives W0x, B × T × 768. The side path first computes Ax, only B × T × 8, then B(Ax), back to B × T × 768. The two outputs are added. The side path never builds the full 768 × 768 matrix.

How much of a change fits in rank r?

The left grid is a "true" 16 × 16 weight change. The middle grid is the best rank-r version of it; the right grid is what is left over. Slide r, and slide how spread out the change is.

2
focused

The true change is built as U·diag(σ)·VT with random orthogonal U and V and σi = e−i/τ, so its singular values are known exactly. By the Eckart–Young theorem, keeping the top r of them is the best rank-r approximation.

The error you see follows one rule. Write the change by its singular values σ1 ≥ σ2 ≥ …, the strengths of its independent directions. The best rank-r version keeps the top r and drops the rest. The share it misses is:

err(r)
The size of the leftover change, as a share of the whole change.
Σi>r σi2
The energy in the directions rank r cannot hold.
Σi σi2
The energy of the whole change.

Worked numbers: singular values 1, 0.5, 0.25, 0.125 and nothing else. Rank 2 misses 0.252 + 0.1252 = 0.078 of a total 1 + 0.25 + 0.0625 + 0.0156 = 1.328. err = √(0.078 / 1.328) = 0.24, so rank 2 holds most of the change. If the singular values decay fast, a small rank is enough. Chapter 5 asks whether real fine-tuning changes decay fast.

LoRA with rank r = 8 on a 768 × 768 matrix trains how many numbers?

Chapter 4

LoRA's small print: zero init, α/r, and merging

Train a LoRA patch live and see why B starts at zero, what the α/r scale is for, and how merging removes all inference cost

The full LoRA forward pass has one more piece, a fixed scale:

W0x
The frozen pretrained path. Never updated.
A
Starts random (the paper uses a Gaussian). It receives gradients.
B
Starts at zero, so BA = 0 at step 0. It receives gradients.
α/r
A constant scale. α is fixed; dividing by the rank r keeps the size of early updates about the same when you change r.
h
The layer's output.

All three details are in the paper's method section: random Gaussian A, zero B, so ΔW = BA is zero at the start, and the update scaled by α/r Hu 2021 · §4.1. The authors set α to the first r they tried and did not tune it, because with Adam, tuning α is roughly like tuning the learning rate Hu 2021 · §4.1.

Why zero B? At step 0 the network computes exactly what the pretrained network computes. Training starts from what the model knows, not from a random perturbation of it. Why not zero A too? Because then both gradients would be zero forever. The gradient of B is proportional to Ax, and the gradient of A is proportional to B. With B = 0 and A random, B moves first, and then A starts moving.

Train a LoRA patch

A frozen 16 × 16 layer must learn a rank-3 change. Four LoRA patches train at once, ranks 1, 2, 4 and 8, with Adam. Switch the two details off, one at a time, and press Train.

Computed in this page: 64 random inputs, mean squared error to the target outputs, Adam with learning rate 0.01, α = 4, 200 steps. The dashed line is the loss of the frozen layer with no patch, the "pretrained model".

Three things to see. With B at zero, every curve starts on the dashed line: step 0 is the pretrained model. With B random, the curves start above it: the patch first damages the model, then repairs it. With the α/r scale on, the four ranks start down the curve together. Schulman and Thinking Machines report the same thing for real LLMs: with the 1/r factor, "the learning curve is exactly the same at the beginning of training, regardless of rank" LoRA Without Regret · 2025. Rank 1 and 2 then stall above zero, because the target change has rank 3.

Merging: zero cost at inference

After training, compute W = W0 + (α/r)BA once, and store it. The model is now an ordinary network with the same shapes as before. The paper stresses this: LoRA adds no inference latency "by construction" Hu 2021 · §4.1. To switch tasks, subtract BA and add another task's B′A′. Press Merge above to check that the merged matrix gives the same outputs.

The savings are large at scale. On GPT-3 175B, LoRA cut training memory from 1.2 TB to 350 GB, and with r = 4 on the query and value matrices cut each checkpoint from 350 GB to 35 MB Hu 2021 · §4.2. Training ran 25% faster than full fine-tuning, because most gradients are never computed Hu 2021 · §4.2.

Why is B initialized to zero in LoRA, while A is random?

Chapter 5

Why low rank is enough: intrinsic dimension

Train the whole network inside a random subspace of d directions, find the smallest d that works, and read the evidence from real language models

LoRA rests on a bet: a good fine-tune lives in a small space. Can we measure how small? Aghajanyan, Zettlemoyer and Gupta did, in 2020 Aghajanyan 2020 · §3.

The trick is to forbid the model from moving freely. Pick d random directions in the space of all D weights. Let training move the weights only along those directions:

θ0
All D pretrained weights, flattened into one long vector. Frozen.
P
A fixed random D × d matrix: d random directions in weight space.
z
The only thing that trains: d numbers, one per direction. It starts at zero, so θ = θ0 at step 0.
θ
The weights the network actually uses.

The intrinsic dimension d90 is the smallest d that reaches 90% of the score full fine-tuning gets Aghajanyan 2020 · §3. Worked example from the paper's own definition: if full fine-tuning reaches 85% accuracy, d90 is the smallest d that reaches 0.9 × 85% = 76.5% Aghajanyan 2020 · §3.

Measure the intrinsic dimension

The hero's network has 561 weights. The task needs real adaptation (domain gap 0.6, 256 labels). Press Run: it fine-tunes inside random subspaces of 1, 2, 4, … directions. Then slide to read each result.

16

Computed in this page: 250 Adam steps on z for each d, one seed, P Gaussian with variance 1/D. The head is part of θ and starts at zero. Your browser's numbers are the toy's own, not the paper's.

The real numbers are striking. On the MRPC paraphrase task, RoBERTa-Large reached 90% of full fine-tuning with about 200 trainable numbers out of some 355 million Aghajanyan 2020 · abstract. Their Table 1 lists d90 = 207 for MRPC and 774 for QQP with RoBERTa-Large, and 1,608 for MRPC with the smaller BERT-Base Aghajanyan 2020 · Table 1. They also found that pretraining lowers intrinsic dimension, and that larger models tend to have lower intrinsic dimension Aghajanyan 2020 · abstract.

That is the "why" behind LoRA. If a few hundred directions in the whole weight space are enough, then a rank-8 change per matrix is generous. The LoRA paper saw this directly: on GPT-3 175B, adapting the query and value matrices with r = 1 scored 73.4 on WikiSQL and 91.3 on MultiNLI, against 73.5 and 91.4 with r = 64 Hu 2021 · Table 6.

Full fine-tuning reaches 80% accuracy. Training in a random subspace of 100 directions reaches 73%, and of 200 directions 75%. What is d90?

Chapter 6

Forgetting and distorted features: LP-FT

See how a random head pushes the backbone in useless directions, why that hurts shifted data and old skills, and how probing first fixes it

Two things can go wrong when every weight is free to move. They have the same cause.

Catastrophic forgetting: the network gets better at the new task and worse at what it could do before. Howard and Ruder named the risk directly: "Overly aggressive fine-tuning will cause catastrophic forgetting" Howard & Ruder 2018 · §3.3.

Feature distortion: the network gets better on data like its training set, the in-distribution (ID) data, and worse on shifted data, the out-of-distribution (OOD) data. Kumar, Raghunathan, Jones, Ma and Liang showed this in 2022 Kumar 2022 · abstract.

What Kumar et al. found

They compared full fine-tuning (FT) and linear probing (LP) on 10 distribution shifts, from CIFAR-10 to STL, to ImageNet sketches and satellite images, using good pretrained features (MoCo-v2 and CLIP) Kumar 2022 · §4. Averaged over the shifts:

MethodID accuracy (avg of 6)OOD accuracy (avg of 10)
Linear probe (LP)82.9%66.2%
Full fine-tune (FT)85.1%59.3%
Probe, then fine-tune (LP-FT)85.7%68.9%

From Kumar et al. 2022, Tables 1 and 2. Linear probing beat fine-tuning OOD on 8 of the 10 shifts; the two exceptions, CIFAR-10.1 and ImageNetV2, are shifts built to closely copy their ID data Kumar 2022 · §4.1.

Fine-tuning won ID by about 2 points and lost OOD by about 7. On ImageNet-R (renditions such as art and cartoons), FT got 52.4% against LP's 70.6% Kumar 2022 · Table 2. And this was not plain overfitting. Even when they picked the fine-tuning checkpoint by OOD accuracy, which you cannot do in practice, LP still won, 67.1% against 61.3% Kumar 2022 · §4.3.

Why a random head distorts features

Look at the gradient that reaches the backbone. The head scores a feature z with a weight vector v: score = v·z. With a logistic loss, the gradient with respect to the feature is:

∂L/∂z
How the loss wants the feature to move. Backpropagation turns this into updates for every backbone weight.
p
The predicted probability of class 1.
y
The true label, 0 or 1.
v
The head's weights. At step 0 of fine-tuning they are random.

Worked numbers: at step 0 the random head is unsure, p ≈ 0.5. For a class-1 example, p − y = −0.5. So every feature is pushed along −0.5·v, a random direction chosen by a random head. The backbone bends its features to suit a head that knows nothing yet. It bends them only for the ID training inputs. Shifted inputs, which training never sees, are left behind with features that no longer match the head Kumar 2022 · §1.

The fix follows from the formula. Train the head first, with the backbone frozen: a linear probe. Now p ≈ y on most examples, so (p − y) is small, and the backbone gets small, meaningful gradients. Then fine-tune everything. This is LP-FT. In the paper, LP-FT changed features about 20 times less than fine-tuning did, on both ID and OOD inputs Kumar 2022 · §4.3.

Probe, fine-tune, or both?

The hero's network, 48 labels. Three methods train side by side. Raise the size of the random starting head and watch fine-tuning's features drift. Raise the domain gap to see when fine-tuning's extra freedom is needed.

0.7
0.00

Each bar is the mean of three seeds. "Old skills kept" runs the three original pretraining heads on the new features. "Feature drift" is the mean distance between each test point's feature before and after training, the same measurement Kumar et al. use. Illustrative toy numbers.

In LP-FT, why does training the head first reduce feature distortion?

Chapter 7

Different speeds for different layers

Give lower layers smaller learning rates, unfreeze from the top down, and see the trade between protecting old features and learning new ones

Freezing is all or nothing: a weight either moves or it does not. There is a softer dial. Let every layer train, but at a different speed.

Howard and Ruder introduced this in ULMFiT in 2018, under the name discriminative fine-tuning Howard & Ruder 2018 · §3.2. Each layer l gets its own learning rate ηl. Their recipe: choose the rate for the last layer by fine-tuning the last layer alone, then divide by 2.6 for each layer below:

ηl
The learning rate of layer l. The top layer gets the largest.
ηl−1
The rate of the layer just below: smaller.
ξ
The decay factor per layer. ULMFiT found 2.6 to work well Howard & Ruder 2018 · §3.2.

Worked numbers for a 12-layer model with η12 = 10−4: layer 11 gets 3.8 × 10−5, layer 6 gets 10−4 / 2.66 = 3.2 × 10−7, and layer 1 gets 10−4 / 2.611 = 2.7 × 10−9. The bottom layers barely move. That is the point: they hold the most general features.

Slow down the lower layers

Full fine-tuning of the hero's network, 48 labels. Layer 2 trains at rate/ξ and layer 1 at rate/ξ2. ξ = 1 is plain fine-tuning. Move ξ, then move the domain gap.

2.6
0.00

Mean of three seeds, 300 steps of SGD with momentum. Ghost marks show plain fine-tuning (ξ = 1). Illustrative toy numbers.

ULMFiT pairs this with gradual unfreezing: first train only the last layer for one epoch, then unfreeze the next layer down and train again, and so on, until all layers train Howard & Ruder 2018 · §3.3. Both ideas say the same thing as LP-FT: settle the top before you disturb the bottom. With these tricks, ULMFiT with only 100 labelled examples matched training from scratch on 100 times more data Howard & Ruder 2018 · abstract.

With ULMFiT's factor 2.6 and a top-layer rate of 10−3, what rate does the layer two below the top get?

Chapter 8

Build it in PyTorch

Freeze with requires_grad_, write a LoRA layer from scratch in about 20 lines, inject it into a ViT, then do the same with one peft call

Everything in this lesson comes down to one flag per weight. In PyTorch every parameter has requires_grad. If it is False, autograd computes no gradient for it, and the optimizer leaves it alone. The rest is bookkeeping. The lesson PyTorch autograd explains the flag.

1. Freeze, probe, unfreeze the top, set per-layer rates

import torch
from transformers import ViTForImageClassification

model = ViTForImageClassification.from_pretrained(
    "google/vit-base-patch16-224-in21k", num_labels=10)   # 12 layers, d = 768

# Linear probe: freeze everything, then let only the new head train
model.requires_grad_(False)
model.classifier.requires_grad_(True)          # Linear(768 -> 10): 7,690 numbers

# Last-k: also unfreeze the top k blocks and the final norm
k = 2
for block in model.vit.encoder.layer[-k:]:
    block.requires_grad_(True)                 # 7,087,872 numbers each
model.vit.layernorm.requires_grad_(True)

n = sum(p.numel() for p in model.parameters() if p.requires_grad)
print(f"trainable: {n:,}")

# Discriminative rates (ULMFiT): each block down gets rate / 2.6
model.requires_grad_(True)
top_lr, xi = 1e-4, 2.6
head = list(model.classifier.parameters()) + list(model.vit.layernorm.parameters())
groups = [{"params": head, "lr": top_lr}]
for depth, block in enumerate(reversed(model.vit.encoder.layer), start=1):
    groups.append({"params": block.parameters(), "lr": top_lr / xi**depth})
groups.append({"params": model.vit.embeddings.parameters(), "lr": top_lr / xi**13})
opt = torch.optim.AdamW(groups, weight_decay=0.01)

2. LoRA from scratch

A LoRA layer wraps a frozen nn.Linear and adds the side path. Read the shapes in the comments: the side path goes from din down to r and back up to dout.

import math
import torch
import torch.nn as nn

class LoRALinear(nn.Module):
    """y = W0 x + b + (alpha / r) * B A x, with W0 and b frozen."""
    def __init__(self, base: nn.Linear, r=8, alpha=16):
        super().__init__()
        self.base = base.requires_grad_(False)          # W0: (d_out, d_in)
        d_in, d_out = base.in_features, base.out_features
        self.A = nn.Parameter(torch.empty(r, d_in))      # (r, d_in), random
        self.B = nn.Parameter(torch.zeros(d_out, r))     # (d_out, r), zero: BA = 0
        nn.init.kaiming_uniform_(self.A, a=math.sqrt(5))
        self.scale = alpha / r
        self.merged = False

    def forward(self, x):                                # x: (B, T, d_in)
        y = self.base(x)                                 # (B, T, d_out)
        if not self.merged:
            side = (x @ self.A.T) @ self.B.T             # (B,T,r) -> (B,T,d_out)
            y = y + self.scale * side
        return y

    @torch.no_grad()
    def merge(self):                                     # fold the patch into W0
        self.base.weight += self.scale * (self.B @ self.A)
        self.merged = True

Now swap it in. The attention block of a Hugging Face ViT names its projections query, key and value. We replace two of them in every layer:

def add_lora(model, names=("query", "value"), r=8, alpha=16):
    model.requires_grad_(False)                          # freeze everything first
    for module in list(model.modules()):
        for child_name, child in list(module.named_children()):
            if child_name in names and isinstance(child, nn.Linear):
                setattr(module, child_name, LoRALinear(child, r, alpha))
    return model

model = add_lora(model)                                  # 12 x 2 x 12,288 = 294,912
model.classifier.requires_grad_(True)                    # the new head always trains
opt = torch.optim.AdamW(
    [p for p in model.parameters() if p.requires_grad], lr=1e-3)

# after training: merge, and the model is a plain ViT again
for m in model.modules():
    if isinstance(m, LoRALinear):
        m.merge()

3. The library one-liner

The peft library does the same injection, plus saving and loading only the small patch:

from peft import LoraConfig, get_peft_model

config = LoraConfig(r=8, lora_alpha=16, lora_dropout=0.1,
                    target_modules=["query", "value"],  # module names to patch
                    modules_to_save=["classifier"])      # the head trains in full
model = get_peft_model(model, config)
model.print_trainable_parameters()                       # about 0.35% of the model
# ... train exactly as before ...
model.save_pretrained("xray-lora")                       # a few MB, not 340 MB
model = model.merge_and_unload()                         # W = W0 + (alpha/r) B A

Which modules should you patch? The original paper patched only attention weights, and found query plus value best under a fixed budget Hu 2021 · Table 5. Later work changed the default. Thinking Machines found that "attention-only LoRA significantly underperforms MLP-only LoRA", and that adding attention on top of the MLP gave no further benefit LoRA Without Regret · 2025. For language models, patch every linear layer, MLP included. Use the device to see what that costs.

The LoRA bill

Pick a model, which matrices to patch, and the rank. The count is exact: r × (din + dout) per patched matrix.

8

The 8B decoder uses a Llama-3-8B-style shape: d = 4,096, MLP width 14,336 with gate, up and down projections, key and value width 1,024 (grouped-query attention), 32 layers; its total includes embeddings. File size assumes 2 bytes per number.

In the LoRALinear class, what does merge() do, and why is it safe after training?

Chapter 9

Did it work?

Compare against a linear probe, track shifted data, measure which layers changed, read the learning curves, and run the shuffled-label test

A fine-tune can look fine on one number and be broken underneath. Five checks catch most problems. Run them every time.

  1. Beat the probe. Train a linear probe on the same frozen features first. It is cheap and hard to get wrong. If your fine-tune does not beat it on held-out data, the extra freedom bought nothing. Kumar et al. used exactly this comparison Kumar 2022 · §4.1.
  2. Track shifted data. Keep a test set that differs from training: another hospital, another microphone, another season. Report ID and OOD side by side. Chapter 6 showed they can move in opposite directions.
  3. Check the old skills. If the model must still do its old job, keep a small test set for it and measure it after fine-tuning.
  4. Look at which layers changed. For each weight matrix, compute the relative change &lVert;W − W0&rVert; / &lVert;W0&rVert;. Large changes in bottom layers on a small dataset are a warning sign.
  5. Shuffle the labels. Train once with labels randomly permuted. Training accuracy may still climb, because a big model can memorize noise. Held-out accuracy must sit at chance. If it does not, test data is leaking into training.
δl
How much layer l moved, as a share of its pretrained size.
&lVert;·&rVert;F
The Frobenius norm: the square root of the sum of all squared entries. For LoRA, Wl − Wl0 is (α/r)BA.

Worked numbers: a 768 × 768 matrix with entries of typical size 0.02 has &lVert;W0&rVert;F ≈ 0.02 × 768 = 15.4. If fine-tuning moved each entry by about 0.001, &lVert;ΔW&rVert;F ≈ 0.001 × 768 = 0.77, so δ ≈ 5%.

Run the checks

Domain gap 0.5, 96 labels. Pick a method and press Train. Then switch on Shuffle labels and train again.

One seed, 400 steps, accuracy measured every 10 steps on 500 held-out familiar points and 400 shifted points. This tiny toy trains with a large learning rate, so its weights move far more than in a real fine-tune, where a few percent is typical. Illustrative toy numbers.

import copy, torch

ref = copy.deepcopy(model.state_dict())                  # snapshot before training
# ... fine-tune ...

@torch.no_grad()
def weight_change(model, ref):
    """Relative Frobenius change of every 2-D weight (merge LoRA first)."""
    out = {}
    for name, w in model.named_parameters():
        if w.ndim == 2 and name in ref:
            w0 = ref[name].to(w.device)
            out[name] = ((w - w0).norm() / w0.norm()).item()
    return out

for name, d in sorted(weight_change(model, ref).items()):
    print(f"{d:7.4f}  {name}")                           # bottom layers should move least
You train with shuffled labels. Training accuracy reaches 90% and held-out accuracy is 50% on a balanced two-class task. What does this tell you?

Chapter 10

Where else it works, and when not to use it

Follow the same decision through audio, medical imaging, language models, robot policies and satellite images, then read the decision chart

The question "how much to unfreeze" has the same three inputs in every field: how many labels you have, how far the new data is from the pretraining data, and how much compute and latency you can spend. Here is the same block in five fields.

FieldPretrained modelTypical choiceWhy
AudioCLAP or an audio foundation modelFrozen encoder + small head; LoRA if the sounds are unusualEmbeddings already separate many sound events; labels for a rare sound are few. See CLAP and the frozen-probe chapter of AudioFM.
Medical imagesImageNet-trained CNN or ViTFine-tune deeper, or LoRA at higher rank; probe as baselineA big domain gap. Raghu et al. found that on two large medical tasks, ImageNet transfer offered little gain in final accuracy, and small models did as well Raghu 2019 · abstract.
LLM instruction tuningA base language modelLoRA on every linear layer; QLoRA if memory is tightSmall-to-medium datasets; many tasks share one base. QLoRA fine-tuned a 65B model on one 48 GB GPU Dettmers 2023 · abstract.
Robot policiesA vision-language-action modelLoRA on all linear layersOpenVLA: LoRA r = 32 reached 68.2% success against 69.7% for full fine-tuning, training 1.4% of the parameters; last-layer-only reached 30.3% Kim 2024 · Table 1.
Satellite imagesA remote-sensing or ImageNet modelLP-FT; watch region and time shiftOn FMoW, test regions differ from training. Kumar et al.: OOD 32.0% for FT, 36.6% for LP, 36.8% for LP-FT Kumar 2022 · Table 2.

The robot result is worth a second look. OpenVLA's authors found that freezing the vision encoder gave 47.0% success, and that "further adaptation of the visual features to the target scene is crucial" Kim 2024 · §5.3. LoRA on all layers did that cheaply, at 59.7 GB of GPU memory against 163.3 GB for full fine-tuning Kim 2024 · Table 1. The lesson microVLA builds these models.

The decision chart

Each region names the method this lesson recommends. Tap the chart, or pick a field. Then add a constraint and see the chart change.

A judgement chart that summarizes this lesson's evidence; the borders are rules of thumb, not measured lines. Field positions are typical cases, not fixed facts.

When not to use it

AvoidWhenUse instead
Full fine-tuningTiny data on a huge model; you care about shifted data or old skillsA probe, then LoRA or LP-FT. A random head on full fine-tuning distorts features (Chapter 6).
Low-rank LoRAA big domain shift, or a dataset larger than the patch can holdHigher rank on every layer, or full fine-tuning. Thinking Machines: when datasets exceed LoRA's capacity, "LoRA underperforms FullFT" LoRA Without Regret · 2025. See the LoRA Without Regret teardown.
LoRA at very large batchTraining with very large batchesSmaller batches, or full fine-tuning. LoRA "is less tolerant of large batch sizes than FullFT", and higher rank does not fix it LoRA Without Regret · 2025.
AdaptersLatency matters at batch size 1LoRA, merged into the weights: +0 ms against +20.7% to +30.3% for adapters in the LoRA paper's test Hu 2021 · Table 1.
A linear probe aloneThe answer is not in the frozen features (big gap)Unfreeze something. In the hero, the probe stalls near 70% at a large gap however many labels you add.
You serve one model with real-time, batch-size-1 requests. Which adaptation adds no inference latency?

Chapter 11

Connections

The memory bill for each method, a one-screen cheat sheet, the honest limits, and where to go next

One last cost decides many real projects: GPU memory. Full fine-tuning with Adam holds, for every weight, the weight, its gradient and two optimizer moments. PEFT methods hold optimizer state only for the small patch. The LoRA paper's GPT-3 numbers show the scale: 1.2 TB down to 350 GB Hu 2021 · §4.2.

bw
Bytes to store one frozen weight: 2 in 16-bit precision, about 0.5 in 4-bit as in QLoRA.
16
Bytes per trained weight in mixed precision with Adam: a 16-bit weight and gradient, plus a 32-bit master copy and two 32-bit moments.
Mem
Memory for weights and optimizer state. Activations come on top and grow with batch size and sequence length.

Worked numbers for an 8-billion-parameter model: full fine-tuning needs about 16 × 8 × 109 = 128 GB before activations. LoRA with 42 million trained numbers needs 2 × 8 × 109 + 16 × 42 × 106 ≈ 16.7 GB. With a 4-bit base, about 4.7 GB.

The memory bill

Slide the model size. Each bar is the memory for weights and optimizer state, against common GPU sizes.

8B

The formula above; LoRA trains 0.5% of the weights. Activations, the KV cache and framework overhead are not counted, so real runs need more. A rule of thumb, not a measurement.

The cheat sheet

MethodWhat trains (d = 768, 12 layers)Inference costBest when
Linear probeHead only: dC + C (7,690)None extraFew labels, small gap; always the baseline
Last k layersk(12d2 + 13d) + head (14.2M for k = 2)None extraMedium data, gap in high-level features
BitFitBiases only (about 0.1%)None extraSmall to medium data, smallest storage
Prompt / prefixL·d at input, or 2·L·d per layer (184,320 for L = 10)Longer sequenceVery large models, many tasks
Adapters2(2md + d + m) per layer (2.4M for m = 64)Extra layers: slower at batch 1Many tasks, latency not critical
LoRAr(din + dout) per matrix (294,912 for r = 8 on q, v)None after mergingMost cases; patch every linear layer
LP-FTHead, then everythingNone extraEnough data to fine-tune, and shifted data matters
Full fine-tuneEverything (85M)None extra, but one full copy per taskLarge data, large gap, memory to spare

The block in one line: the maths is W = W0 + (α/r)BA with B = 0 at the start; the knobs are what to unfreeze, rank r, α, learning rate (about 10× full fine-tuning for LoRA) and per-layer decay; the failures are forgetting, feature distortion from a random head, too low a rank for a big gap, and adapter latency; the alternatives are a probe, LP-FT, and full fine-tuning.

Honest limits

The toy in this page has 561 weights; real models have billions. Its trends match the papers, but its numbers are its own. The LP-FT theory is proved for two-layer linear networks Kumar 2022 · §3; the deep-network evidence is empirical. Intrinsic dimension was measured with random projections, so the true value is likely lower than reported Aghajanyan 2020 · §4.2. And the decision chart is judgement, not law: when in doubt, run a probe and a LoRA and compare.

Where this sits, and where to go next

The takeaway. Change as little as the task needs. Start with a probe, because it is cheap, safe and the baseline for everything else. Add LoRA when the probe stalls. Train the head before the body. Slow the lower layers. Unfreeze everything only when you have the data, the gap, and the memory to justify it, and always check shifted data and old skills.
About how much memory do weights and Adam state need for full fine-tuning of an 8-billion-parameter model in mixed precision?

Now press Present or Teach and explain, from memory, why fine-tuning every weight can lose to a linear probe on shifted data, and what LoRA's zero-initialized B buys you. Then go back to the hero and find the domain gap where the probe stops being enough.

Sources: Hu et al., "LoRA: Low-Rank Adaptation of Large Language Models", ICLR 2022 (arXiv:2106.09685) · Houlsby et al., "Parameter-Efficient Transfer Learning for NLP", ICML 2019 (arXiv:1902.00751) · Kumar, Raghunathan, Jones, Ma and Liang, "Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution", ICLR 2022 (arXiv:2202.10054) · Aghajanyan, Zettlemoyer and Gupta, "Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning", ACL 2021 (arXiv:2012.13255) · Howard and Ruder, "Universal Language Model Fine-tuning for Text Classification", ACL 2018 (arXiv:1801.06146) · Ben-Zaken, Ravfogel and Goldberg, "BitFit", ACL 2022 (arXiv:2106.10199) · Li and Liang, "Prefix-Tuning", ACL 2021 (arXiv:2101.00190) · Lester, Al-Rfou and Constant, "The Power of Scale for Parameter-Efficient Prompt Tuning", EMNLP 2021 (arXiv:2104.08691) · Schulman and Thinking Machines Lab, "LoRA Without Regret", September 2025 (blog) · Kim et al., "OpenVLA", CoRL 2024 (arXiv:2406.09246) · Dettmers et al., "QLoRA", NeurIPS 2023 (arXiv:2305.14314) · Raghu, Zhang, Kleinberg and Bengio, "Transfusion: Understanding Transfer Learning for Medical Imaging", NeurIPS 2019 (arXiv:1902.07208).
Back to Gleams · The Builder's Toolkit hub