LLaMA-Adapter teaches a frozen 7B language model to follow instructions by training only 1.2 million new numbers, and a gate that starts at zero keeps the model’s old skills intact.
Efficient fine-tuning of language models with zero-initialized attention: how a tiny add-on learns a new job without breaking what the big model already knows.
Picture a song that already sounds right, and a new voice you want to add without spoiling it. The gate is the volume knob. Drag the gate from 0 up. Watch the right panel start exactly on the original answer, while the left panel, a plain add-on, is knocked off it before any training. Then we build, piece by piece, the prompts, the gate, the training and the picture version.
You need a feel for what a vector is (a short list of numbers, here an arrow on a page) and the idea that a model is a stack of layers. We build attention and everything else from zero.
Reading for MIT MMAI: Multimodal AI (Spring 2026, Paul Liang), paired with Lecture 8, Large Multimodal Models. Paper: Renrui Zhang, Jiaming Han, Chris Liu, Aojun Zhou, Pan Lu, Yu Qiao, Hongsheng Li and Peng Gao, ICLR 2024, arXiv 2303.16199. Re-taught here in our own words and devices.
Toy attention head: 4 word tokens and 3 adaption-prompt tokens, each with a 2-number value, so one output is an arrow on a page. The scores and values of the prompts are random draws made in your browser; the maths is the paper’s Equation 7: plain mixes everything in one softmax, the gate multiplies the prompts’ own softmax by tanh(g) and leaves the word softmax alone.
Chapter 0
See what it costs to retrain 7 billion numbers for one new skill, and what the paper does instead
Picture a very well-read assistant who has read most of the internet. Ask it “Tell me about alpacas” and it may not answer. It may continue your sentence, as if the question were the start of a web page. It knows a lot. It has not learned the job of answering.
That is the situation with LLaMA, a family of open language models from Meta. The 7B version has about seven billion parameters. A parameter is one learned number inside the model, like one tiny knob on a huge mixing board. Training sets all the knobs. After training, the model predicts the next word well, but it does not follow orders.
To teach the job, people show the model many pairs of an instruction (“Translate this into Spanish”) and a good response. Training on such pairs is called instruction tuning. It is one kind of fine-tuning: more training, on a small set of new examples, that starts from a model already trained.
The paper builds on Stanford Alpaca. The Alpaca team started with 175 instruction and response pairs written by people. They asked GPT-3.5 to invent many more in the same style, a method called self-instruct, and reached 52,000 pairs. Then they fine-tuned all 7 billion parameters of LLaMA on them. The result behaved much like GPT-3.5.
Alpaca works. The paper’s complaint is the price. Full fine-tuning is, in the authors’ words, time-consuming, computation-intensive, and cumbersome to move to other uses.
Move the selector below. It shows Table 1 of the paper: three ways to turn LLaMA 7B into an instruction follower, trained on the same 52,000 pairs on 8 A100 graphics chips.
Each bar is one method, drawn against Alpaca as 100%. Choose what to compare. The thin yellow sliver is the whole add-on.
Values are Table 1 of the paper: tuned parameters 7B, 4.2M and 1.2M; storage 13G, 16.8M and 4.7M; training time 3 h, 1.5 h and 1 h on 8 A100 GPUs. The text of Section 4.1 says 4.9M for the adapter’s storage, the table says 4.7M; we draw the table. Bar lengths are linear, so the add-on is a sliver by design.
Look at the first view. Alpaca changes 7 billion numbers. LLaMA-Adapter changes 1.2 million. That is 0.017% of the work: about one number in 5,800. The authors report one hour of training instead of three.
Now the quiet cost, storage. Say you want ten different specialists: a lawyer, a coder, a tutor, and seven more. With full fine-tuning each specialist is a full copy of the model, 13 GB each, so 130 GB. With an add-on you keep one 13 GB model and ten files of about 4.7 MB. That is 13 GB plus 47 MB. The paper calls this plug with expertise: swap the small file, change the expert.
The family of tricks for doing this is called parameter-efficient fine-tuning (PEFT). You freeze the big model, which means its parameters are never changed, and you train only a small new piece. This paper adds one such piece, with one clever detail.
The piece is a short list of learnable adaption prompts, new tokens placed inside the top layers of the model. The detail is zero-initialized attention: a gate that starts shut, so on day one the model still behaves like the original LLaMA, and the new knowledge flows in only as the gate opens. You already saw that detail in the instrument at the top of this page.
The authors list four features: only 1.2M learnable parameters, under one hour of fine-tuning, swappable adapters, and an extension that lets the same model answer questions about images. The last chapters of this lesson cover each.
Chapter 1
Follow one new word as it looks back at the words before it, and learn which parts of that step the paper can touch
You are reading “The cat sat on the” and you must guess the next word. To guess, you glance back at the earlier words. “Cat” matters a lot. “On” matters a little. You do not weigh every word equally. A language model does the same glance, many times, and the glance has a name: attention.
The paper changes only this one step inside the model. So we need to see it clearly, with every number. Nothing here is special to LLaMA-Adapter yet. It is the ordinary machinery that the adapter plugs into.
A model reads tokens, which are pieces of text such as a word or a part of a word. Inside the model each token is a feature vector: a list of C numbers that describes the token in its context. For LLaMA 7B, C is 4,096. So one token is 4,096 numbers, and M tokens form a table with M rows and C columns, written M × C.
The model is a stack of N transformer layers. LLaMA 7B has N = 32. Each layer takes the table in and hands a table of the same shape to the next layer. Inside each layer, attention comes first. It lets each token read from the earlier tokens.
Think of a library. You walk in with a question. Every book has a short label on its spine, and a full text inside. You compare your question with the labels, pick the books whose labels match best, and read a blend of their texts.
Attention uses the same three roles. The new token produces a query, which is its question. Every earlier token produces a key, which is its label, and a value, which is its content. Each is made by a linear projection: the token’s feature vector multiplied by a fixed table of numbers. LLaMA learned those tables in pretraining. In this paper they stay frozen.
The model is about to write the word after “The cat sat on the”. Pick the question it asks. The bars show how much it reads from each earlier word.
Illustrative: the scores below are made-up numbers chosen to look like a plausible head; a real LLaMA head has 32 of them per layer, with scores learned in pretraining. The weights are an exact softmax of the scores.
The query is compared with each key by a dot product: multiply the two lists number by number and add up. A big result means the question matches the label. That result is a score. The scores for all earlier tokens form one row. Then the model squeezes the row into weights with the softmax, which makes every number positive and makes the row add up to 1.
A worked row. For “Who is acting?” the five scores are 0.2, 2.2, 0.8, −0.4 and 0.0. Raise e to each: 1.22, 9.03, 2.23, 0.67 and 1.00. They add up to 14.14. Divide each by 14.14 and the weights are 0.086, 0.638, 0.157, 0.047 and 0.071. “Cat” gets 64% of the attention.
weights = [0.086, 0.638, 0.157, 0.047, 0.071], sum = 1.000
Suppose each word’s value were the single number 4, 1, 3, 0, 2. The blend is 0.086×4 + 0.638×1 + 0.157×3 + 0.047×0 + 0.071×2 = 1.60. The new word receives a mixture that leans on “cat”. A final linear projection, Linearo, turns that blend back into a vector of size C.
LLaMA 7B does this in parallel in 32 heads. A head is one copy of the attention step with its own projections, and each works on a slice of 128 numbers (4,096 divided by 32). Different heads ask different questions. Their results are joined and mixed by Linearo.
Also remember that the model writes text one token at a time. To write token M+1, it computes the query only for the newest token and compares it with the keys and values of everything before. The paper’s equations describe exactly this step.
Chapter 2
Place K new tokens in front of the words in the top layers, and count exactly how few numbers that adds
You want to steer a student without rewriting the student’s brain. One way is to hand the student a few extra index cards before each exam question. The cards carry hints. The brain stays as it was. Only the cards are rewritten, again and again, until the hints help.
That is the plan of LLaMA-Adapter. The “cards” are adaption prompts: short lists of new, trainable feature vectors that sit in front of the real word tokens.
Adding trainable tokens to a frozen model is an old idea called prompt tuning. The paper cites two flavours. In one, trainable tokens are added only at the very input, next to the word embeddings (Lester et al., 2021). In the other, called prefix tuning, trainable tokens are added in every intermediate layer (Li and Liang, 2021). In both, a prefix is a block of new tokens placed before the real ones.
LLaMA-Adapter takes the second route, with two choices of its own. It uses the top layers only, and it feeds the prefix through a special gated attention (Chapter 4).
Let the model have N layers. The paper picks the topmost L layers, where L is at most N, and gives each its own prompt table Pl of size K by C. K is how many prompt tokens there are. C is the feature size of the model, 4,096.
At layer l the real tokens are a table Tl with M rows: the instruction and the words written so far. The prompt is joined on top along the token direction:
Every real word can now look back at K extra rows as well as at the earlier words. The rows in Pl are not words from any text. They are numbers that training will shape into whatever helps the model follow instructions.
Layers near the input handle spelling and local patterns. Layers near the output handle meaning and the plan of the answer. The authors say prompting at the last layers can better tune the language representations with higher-level semantics. Their default is N = 32 layers, K = 10, and L = 30 (Section 4.1). The ablation in Chapter 8 tests it.
This is the arithmetic the paper’s headline rests on. Each layer has its own table of K × C numbers. There are L such layers. So the prompts hold K × C × L numbers.
10 × 4,096 × 30 = 1,228,800 numbers ≈ 1.2M
The gates of Chapter 4 add one scalar per attention head per layer: 32 × 30 = 960 more. That is too small to change the count. Stored as 4-byte numbers, 1,228,800 values take 4,915,200 bytes. That is 4.7 in units of 1,048,576 bytes, and 4.9 million bytes in decimal units. Both figures appear in the paper, in Table 1 and in the text of Section 4.1.
Try the sizes yourself. Drag K and L and watch which layers are touched and how the count moves.
Left: the 32 layers of LLaMA 7B, blue where frozen, pink where a prompt table is inserted. Right: the prompt rows placed in front of the word rows. Drag K and L.
Exact arithmetic: K × C × L with C = 4,096, compared with 7 billion as the paper rounds LLaMA 7B. The default K = 10, L = 30 is the paper’s setting (Section 4.1); the other settings are for you to try and were not all trained in the paper. Gates (one per head per layer) are left out of the count.
At K = 10 and L = 30 the count is 1,228,800. Cut L to 10 and it falls to 409,600. Raise K to 50 and L to 32 and it reaches 6,553,600, still under 0.1% of the model. Whatever you choose, the cost grows in a straight line with K and with L, never with the size of the frozen model.
Each layer owns a separate Pl. A hint written for layer 5 would not mean much to layer 25, because the feature vectors there describe different things. So training learns 30 independent prompt tables.
Chapter 3
Measure how much attention untrained prompts steal from real words, and why that damages a model that already works
A pianist plays a piece perfectly. You slip ten sheets of scribbles between the pages of the score and say, “Play as before, but learn from these.” On the first bar the pianist glances at the scribbles, and the music goes wrong. The pianist could learn, but the first notes already hurt.
The same thing happens to LLaMA when you add prompts that start as random numbers. This chapter measures the damage. The next chapter fixes it.
Recall Chapter 1. A new word makes a row of scores, one per token it can read. Softmax turns the row into weights that add up to 1. LLaMA was trained with only word tokens in that row. Every head in every layer expects its weights to be split among words.
Now add K prompt rows. They enter the same row and the same softmax. The weights still add up to 1, so the prompts take their share from the words. If the prompt scores are about as large as the word scores, a simple count says how much they take. With K prompts and M + 1 word tokens, each token gets about the same weight, so the prompts get K divided by K + M + 1.
K = 10, M + 1 = 26 words: 10 / (10 + 26) = 28% of the attention goes to the prompts
For a short instruction the share is worse. With M + 1 = 6, it is 10 / 16, which is 62%. In the right head, the real words are ignored more than half the time, and what replaces them is random.
First, the real words lose weight. Their weights are rescaled by one minus the prompt share, so the careful recipe LLaMA learned is diluted. Second, the weight that goes to the prompts reads their values, and a random value adds a random vector to the blend. The head’s output is pushed in a direction nobody chose.
The push does not stay in one place. There are 30 inserted layers. The output of one layer is the input of the next, so an error made in layer 3 is read by layers 4 to 32. The authors state this plainly: randomly initialized prompts bring disturbance to the word tokens at the beginning of training, which harms fine-tuning stability and effectiveness.
Each setting is averaged over 400 random toy heads. The top bar splits attention between prompts (pink) and words (blue). The lower bar is how far the output moves from the original. Change K, the number of words, or how loud the prompt scores are.
Toy simulation, not from the paper: word scores are random with spread 1, values are random 2-number vectors, 400 seeded trials. Shift is the length of the output change divided by the typical length of the original output. A gate closed at g = 0 would show 0% on both bars, by construction (Chapter 4).
Start at the defaults: K = 10, 26 words, scores of size 1. About 27% of the attention goes to prompts, close to the simple count of 28%, and the output moves by about 43% of its own length. Now set the prompt score size to 0. Every prompt scores exactly 0, an average score, the share drops to about 20% and the shift to 25%. Raise the size to 3 and a few prompts get huge scores. They grab about 69% of the attention and the shift passes 160%.
Shrink K to 1 and raise the words to 60. The damage drops, because one prompt among 60 words is a small voice. This is why a tiny prompt at the very start of training might be tolerable, and ten of them at 30 layers is not.
Symptom: training begins from a model that is already worse than the one you froze. The loss starts high, falls slowly, and the model learns to answer badly.
Measurement: the paper’s random-initialized baseline reaches 40.77% accuracy on the ScienceQA validation set. Picking answers at random scores 39.83% (Table 2). So the baseline learned almost nothing useful. In the loss curves of Figure 7 it slowly approaches 0.15 and has not fully converged.
Fix: make the prompts invisible at the start. A prompt that gets zero influence cannot disturb anything. Then let the influence grow. That is the next chapter.
Chapter 4
Split the softmax in two and put a gate that starts closed on the prompt half
Imagine a radio with a volume knob. You want to add a new voice to a song that already sounds right. If the knob starts at the middle, the new voice is loud on the first second and spoils the song. If the knob starts at zero, the song is untouched, and you raise the new voice slowly while you listen.
This is the paper’s central trick. The new voice is the attention that goes to the adaption prompts. The knob is a learnable number called the gate, g. It starts at zero.
The new word tl has its query, and it compares it with keys of the prompts, the earlier words and itself, K + M + 1 tokens in all. Equations 2 to 5 give a row of K + M + 1 scores. The paper then splits that row into two named parts (Equation 6):
A plain model would run one softmax over the whole row. LLaMA-Adapter runs two, one on each part, and multiplies the prompt part by a gate (Equation 7):
Two design decisions sit in this line. First, the separate softmaxes. The words’ softmax never sees the prompt scores, so it equals exactly what the original LLaMA computed. The paper says this preserves the original probability distribution. Second, the gate on the prompt side only. Nothing multiplies the words’ weights, so the pretrained knowledge is not disturbed.
Equation 8 multiplies the weights by the values and applies Linearo. Because the weights are two separate groups, the blend splits too:
blend = (words’ weights · word values) + tanh(g) × (prompts’ weights · prompt values)
The first term is the original LLaMA blend, unchanged. The second term is the prompts’ contribution, scaled by tanh(g). Close the gate and only the old answer is left. That is the whole mechanism, and the next device lets you check it with numbers.
Three word tokens and two prompts. Top row: one softmax over all five. Middle row: two softmaxes, prompts scaled by tanh(g). Bottom line: the blended answer. Drag g.
Worked example, illustrative numbers: word scores 1, 0, −1 with values 3, 1, 2; prompt scores 0.5, −0.5 with values 10, −4. The original answer is 2.42; the plain joint softmax gives 3.78 before training; the gated version gives 2.42 + tanh(g) × 6.24. All from the paper’s Equations 6 to 8.
Check the numbers by hand. Words: softmax of (1, 0, −1) is (0.665, 0.245, 0.090), and the blend is 0.665×3 + 0.245×1 + 0.090×2 = 2.42. Prompts: softmax of (0.5, −0.5) is (0.731, 0.269), and the blend is 0.731×10 + 0.269×(−4) = 6.24. At g = 0 the answer is 2.42 + 0 = 2.42, the original. At g = 1, tanh(1) = 0.762, so the answer is 2.42 + 0.762×6.24 = 7.17.
The plain version, a single softmax over all five scores, gives (0.429, 0.158, 0.058, 0.260, 0.096). Its words have lost a third of their weight, and it answers 3.78 at step zero. That is 56% above the original, with no training at all.
In practice the paper uses a separate g for every attention head, so each of the 32 heads learns its own volume. Some heads may open fast and others stay shut. The authors say this benefits the learning diversity of the multi-head mechanism.
Chapter 5
Trace Equations 1 to 8 through one inserted layer, tensor by tensor, with the real LLaMA 7B sizes
A recipe is easy to nod at and hard to cook. Here we cook it. The model has read an instruction and a few words of reply, 25 words in all. It is about to write the next one. We follow that one new token through one inserted layer, and write down the shape of every table that appears.
The settings are the paper’s: C = 4,096, K = 10 prompt rows, and M word rows already read. Take M = 25.
Step through it below. Drag the number of words and watch which shapes change and which stay the same.
Press Next to light the stages in order. Pink marks the learned table and gate, blue marks frozen LLaMA. The right column is the shape at that stage.
Shapes follow the paper’s Equations 1 to 8 with C = 4,096 and K = 10. The paper writes the scale as the square root of C; a multi-head implementation divides by the square root of each head’s own slice, 128 numbers, and we describe the single-head reading. M is yours to set; the paper’s own M is the instruction plus the reply so far.
Two things to notice. First, only the stack and the keys and values grow when the text grows. The query stays 1×4096, and the output stays 1×4096. Second, the prompt rows travel through the frozen key and value projections of LLaMA. They are not given a new language. They must learn to be numbers that LLaMA’s own projections turn into useful keys and values.
The prompts never ask questions. They are rows to be read, not tokens that read. Only real tokens produce queries. So the prompts add K rows to the key and value tables and nothing else. The extra cost per layer at answer time is small: K more rows to compare, 10 against about 26 in this example.
After training, Pl is fixed. So the prompt part of the keys and values is the same for every request, a fact one could use to save work. The paper does not claim this; it follows from the equations.
Three slips are common. First, a shape slip: if the score row has 35 entries instead of 36, the new token was left out of the stack, and it cannot read itself. Second, a split slip: if you run one softmax and then cut the row in two, the word weights no longer equal the original, and the gate loses its promise. Third, a freeze slip: if any LLaMA table still has gradients switched on, you train far more than 1.2M numbers.
Here is the same layer in Python, for one new token. It uses PyTorch. Each comment names the equation. The matrices Wq, Wk, Wv and Wo are LLaMA’s own and are never updated. Only P and gate are trained.
python import torch, torch.nn.functional as F def zero_init_attention(t, T, P, Wq, Wk, Wv, Wo, gate, heads=32): # t: (1, C) newest token T: (M, C) earlier tokens P: (K, C) adaption prompt # Wq, Wk, Wv, Wo: frozen (C, C) gate: (heads,) learnable, starts at zero C = t.shape[-1]; d = C // heads; K = P.shape[0] X = torch.cat([P, T, t], dim=0) # (K+M+1, C) Eq. 1 q = (t @ Wq).view(1, heads, d) # (1, H, d) Eq. 2 k = (X @ Wk).view(-1, heads, d) # (K+M+1, H, d) Eq. 3 v = (X @ Wv).view(-1, heads, d) # (K+M+1, H, d) Eq. 4 s = torch.einsum('qhd,nhd->hqn', q, k) / d ** 0.5 # (H, 1, K+M+1) Eq. 5 sK, sW = s[..., :K], s[..., K:] # Eq. 6: prompts | words wK = F.softmax(sK, dim=-1) * torch.tanh(gate).view(heads, 1, 1) # Eq. 7 wW = F.softmax(sW, dim=-1) # Eq. 7: words, untouched w = torch.cat([wK, wW], dim=-1) # (H, 1, K+M+1) out = torch.einsum('hqn,nhd->qhd', w, v).reshape(1, C) return out @ Wo # (1, C) Eq. 8
Set gate to zeros and wK becomes all zeros. Then out is exactly what a plain LLaMA head computes, because wW is a softmax over only the words. You can test that claim with one assert, and it is a good first unit test for anyone who re-implements the paper.
Chapter 6
Freeze the giant, send gradients to the prompts and gates, and watch why the gate opens first
You tune a guitar by turning pegs while you listen. You do not rebuild the guitar. Training the adapter is like that: the instrument is frozen, a few pegs move, and the loss is the ear that says “closer” or “further”.
The whole of LLaMA is frozen: embeddings, all N = 32 layers, every Linear table, the output head. Four kinds of numbers learn, and only these:
Training uses the same 52,000 instruction and response pairs as Alpaca. The paper’s settings (Section 4.1): 8 A100 graphics chips, 5 epochs (an epoch is one pass over the data), 2 warm-up epochs (the learning rate rises slowly at first), batch size 64, learning rate 0.009 and weight decay 0.02. The paper does not spell out the loss. The standard choice for instruction tuning, and the one Alpaca uses, is to predict each next word of the response and penalise wrong guesses. We assume that here.
Look at the “Backward” arrow in the paper’s Figure 2. The error signal must travel from the loss at the top, back down through frozen layers, to reach the prompts in layer 3. Each frozen layer passes the signal along but does not store an update for its own weights.
The saving is in what is kept. Many optimizers keep extra numbers for every trained parameter. With 7 billion trained parameters that is a huge load. With 1.2 million it is tiny. The paper adds a point for several machines: only the gradients of 1.2M parameters must be sent between machines, not 7B. The forward and backward pass through the frozen model still costs real compute, which is why training takes one hour and not one minute.
At the start every g is 0. Recall from Chapter 4 that the answer is: original blend + tanh(g) × prompt blend. Ask how the loss changes if each part moves a little. Two facts come from the chain rule.
So the first updates move only the gates. As soon as g leaves zero, tanh(g) is no longer zero, gradients reach the prompts, and the prompts begin to learn. This is the “gate opens first, then the prompts follow” story. The paper puts it as: zero gating first eliminates the influence of under-fitted prompts, then increases its magnitude to provide more instruction semantics. The derivation above is our reading of that sentence.
Watch it happen in a tiny version. The toy has 3 prompt rows, 2-number values and a frozen head, and its target answer differs from the frozen answer. Press Train, or drag the step.
Three curves over 60 steps of gradient descent: the loss (grey, as a share of its start), the gate tanh(g) (orange), and how hard the error pushes on the prompts (pink). Press Train to play it, or drag the step.
A toy, not the paper’s run: one head, 4 frozen word tokens, 3 trainable prompt rows (scores and 2-number values), one gate, squared-error loss to a made-up target, plain gradient descent at a rate of 0.1 with exact numeric gradients. Computed live in your browser. The real model has 30 layers, 32 gates per layer and a language loss.
With the first random draw, the story runs like this. At step 0 the loss equals the frozen model’s loss, and the prompt gradient is exactly 0 while the gate gradient is not. By step 10 the gate has opened about two thirds of the way and the prompt gradient is at its peak. By step 40 the loss is near 0, the prompt gradient has faded and the gate has settled. Draw new prompts and the timing shifts, but the order stays the same. The gate is a handle for the first moves, and it stays open at a level the data wants.
Symptom: a prefix model whose first step already hurts the loss and trains slowly. Measurement: compare the loss curve and a held-out score of the zero-init run with a random-init run, as the paper does in Figure 7 and Table 5. Fix: keep the structure of Chapter 4. Chapter 8 gives the measured numbers.
Chapter 7
Turn one image into one token, add it to every prompt, and train in two stages
You show a friend a photo and ask, “What place is this?” Your friend cannot see a photo in text. But if you whisper a one-line summary of the photo into your friend’s ear, in your friend’s own language, your friend can answer. The multi-modal LLaMA-Adapter does exactly that: it whispers one summary token to the model.
The nice part is that it needs no new machinery. The summary token rides in on the prompts you already know, so it passes through the same gate.
A visual encoder is a model that turns an image into numbers. The paper uses CLIP, which was trained on pairs of images and captions (see our CLIP lesson). It stays frozen. CLIP gives multi-scale global features: several vectors Im, each of size 1 × Cm, each a summary of the whole image at a different level of detail. The paper calls the number of scales M, which clashes with the M words of the earlier chapters, so below we say “the scales”.
The scales are joined into one long vector and passed through a small trained network, the projection. The paper says it is a simple bottleneck MLP, meaning a small network that squeezes the vector and expands it again. Its output is one vector of size 1 × C, the same size as a word token (Equation 9).
Now the key step. Copy Ip K times to make a K × C table, and add it, element by element, to the prompt table of every inserted layer (Equation 10):
Nothing else changes. Plv goes into the zero-initialized attention of Chapter 4 in place of Pl. Its keys and values are made by the same frozen projections, and its attention is multiplied by tanh(g). So the image reaches the answer only through the gate. At g = 0 the model ignores the picture and acts like plain LLaMA. As the gates open, the picture’s influence grows.
For ScienceQA, a question set with images, the authors train on its own training set in one step. They freeze CLIP and LLaMA and train the projection and the zero-init attention. That gives 1.2M (prompts and gates) plus 0.6M (projection) = 1.8M learnable parameters, as Table 2 lists.
For the open-ended tests (MME, MMBench, LVLM-eHub) they use two stages. Stage 1 aligns pictures with words: they use image and caption pairs from LAION-400M to tune the projection and the zero-init attention. Stage 2 teaches answers: they freeze the projection and tune the zero-init attention on Alpaca data mixed with LLaVA-I instruction data. Walk through the three setups below.
Choose a setup. Each row is a part of the model. Pink rows train in that setup; blue rows are frozen; grey has no weights.
From Section 3.3 and Section 4.2. The 1.2M, 0.6M and 1.8M figures are the paper’s (Section 4.2, Table 2). Counts for the two stages follow from which parts the text says are tuned in each stage. The paper does not give a separate count for each stage.
In stage 2 only the 1.2M of prompts and gates learn. The projection that was aligned in stage 1 is frozen so that the instruction data cannot undo the alignment. That is our reading of why the authors split the stages. The paper states the split, not the reason.
Chapter 8
Read each main table and figure, compare the ablations, and see where the evidence stops
A claim is only as good as the table behind it. You now know the mechanism. This chapter reads the paper’s evidence the way a careful reviewer would: what was measured, against what, and what the number does and does not say.
Almost every result uses LLaMA 7B with N = 32 layers, K = 10 and L = 30, fine-tuned on the 52,000 Alpaca pairs for language tests (Section 4.1). The multi-modal tests use CLIP as the eye and a bottleneck MLP as the projection (Section 4.2). Choose a view below to see each main number drawn on one axis.
Pink bars are LLaMA-Adapter. Blue bars are other methods. Red marks the random-initialized baseline. Pick a table and read the sentence under it.
Efficiency (Table 1). 1.2M tuned parameters, 4.7M storage and 1 hour on 8 A100 chips, against 7B, 13G and 3 hours for Alpaca, and 4.2M, 16.8M and 1.5 hours for Alpaca-LoRA. This is the headline and it is measured directly.
Instruction quality (Figure 4, Figure 5, Table 18). Side-by-side answers for question answering, translation and code look similar to Alpaca’s. In the GPT-4 judging benchmark, where GPT-4 compares two answers on 80 questions, LLaMA-Adapter wins more often than Alpaca and than Alpaca-LoRA. On the Open LLM benchmark (ARC, HellaSwag, MMLU, TruthfulQA) the averages are 52.2 for LLaMA-Adapter, 50.73 for Alpaca-LoRA and 49.23 for Alpaca.
Pictures (Table 2, Table 3). On ScienceQA the text-only variant reaches 78.31% and the multi-modal one 85.19%, a gain of 6.88 points from the 0.6M projection. For comparison the table lists human 88.40, GPT-4 with chain of thought 83.99 and MM-CoT 84.91. On MME perception it scores 973, against 867 for MiniGPT-4 and 503 for LLaVA. On MME cognition it scores 249, below MiniGPT-4’s 292.
An ablation removes or changes one part to see what breaks. Two ablations matter here, both on the ScienceQA validation set.
How many layers (Table 4). With 10 layers the score is 55.95%; with 20, 73.36%; with 30, 83.85%; with all 32, 81.03%. The parameter counts are 0.97M, 1.37M, 1.79M and 1.83M. More layers help up to a point. Too many, the authors say, disturb the early encoding of the input words. They also note that with few resources, inserting into all layers is generally a good solution.
The gate itself (Table 5, Figure 7). Random initialization scores 40.77%. Zero initialization scores 83.85%, a gain of 43.08 points. Random choice on this test scores 39.83%, so the random-init run is almost at chance. In the loss curves the zero-init run falls faster at the start and converges toward zero, while the random-init run slowly approaches 0.15.
This is the strongest evidence in the paper. The same prompts, the same data, the same training time, and one change: how the attention starts.
Section 4.4 applies the same attention to ViT on VTAB-1k, RoBERTa-large on SQuAD, and CLIP on base-to-novel tests. The text reports better results than full fine-tuning on VTAB-1k and the best average on the CLIP test. On SQuAD 2.0, zero-init (87.2 F1) trails full fine-tuning (89.4) and beats the stronger prompt-tuning baseline, P-Tuning v2 (85.5). So the claim is “competitive and cheap”, not “always best”.
Chapter 9
Keep the numbers, the terms, a checklist and the core code on one page
Here is the whole paper in one paragraph. Full fine-tuning of a 7B model for instructions is slow and heavy. LLaMA-Adapter freezes the model and adds K = 10 learnable prompt rows in each of the top L = 30 layers, which is 1.2M new numbers. The prompts enter attention through a split softmax, and their share is scaled by tanh(g) with g started at zero, so training begins from the original model and lets instruction knowledge in as the gate opens. One hour on 8 A100 chips gives answers comparable to Alpaca. Adding one CLIP-based image token to every prompt turns the same model into a picture-and-text model.
| Quantity | Value | Where |
|---|---|---|
| Layers N, prompts K, inserted layers L | 32, 10, 30 | Section 4.1 |
| Feature size C | 4,096 (LLaMA 7B) | Chapter 2 arithmetic |
| Learnable parameters | 1.2M (10 × 4,096 × 30 = 1,228,800) | Abstract, Table 1 |
| Multi-modal learnable parameters | 1.8M (1.2M + 0.6M projection) | Table 2, Section 4.2 |
| Storage per adapter | 4.7M in Table 1, 4.9M in the text | Table 1, Section 4.1 |
| Training data and schedule | 52K pairs, 5 epochs, 2 warm-up, batch 64, rate 0.009, decay 0.02 | Section 4.1 |
| Training time | 1 hour on 8 A100 chips (Alpaca: 3 hours) | Table 1 |
| ScienceQA average | 85.19% (text only: 78.31%) | Table 2 |
| Zero-init gain | 40.77% to 83.85%, +43.08 points | Table 5 |
| Layers inserted | 30 layers 83.85%, all 32 layers 81.03% | Table 4 |
| MME perception, cognition | 973, 249 | Table 3 |
| Term | Plain meaning | Chapter |
|---|---|---|
| Adaption prompt Pl | K learned rows, one table per inserted layer, placed before the word rows | 2 |
| Zero-initialized attention | Attention with two softmaxes and a gate that starts at zero on the prompt side | 4 |
| Gate gl | A learned number per head, squeezed by tanh to the range −1 to 1 | 4 |
| Frozen | Never updated by training; gradients still pass through it | 6 |
| Image token Ip | One vector summarising the image, same size as a word token | 7 |
| Ablation | A test that removes or changes one part to see its effect | 8 |
python import torch, torch.nn as nn K, L, C, HEADS = 10, 30, 4096, 32 llama = load_llama_7b() # your loader for p in llama.parameters(): p.requires_grad = False # freeze all 7B prompts = nn.ParameterList([nn.Parameter(torch.randn(K, C) * 0.02) for _ in range(L)]) gates = nn.Parameter(torch.zeros(L, HEADS)) # zero-init: the whole trick # the paper does not give the prompt init; the closed gate makes the start safe either way # swap attention in the top 30 of 32 layers for zero_init_attention (Chapter 5) opt = torch.optim.AdamW([*prompts.parameters(), gates], lr=0.009, weight_decay=0.02) # optimizer name is our assumption; the paper gives the rate and decay only assert sum(p.numel() for p in prompts.parameters()) == 1_228_800 for batch in alpaca_52k: # 5 epochs, batch 64, 2 warm-up epochs loss = response_next_word_loss(llama, batch) # assumed standard LM loss loss.backward(); opt.step(); opt.zero_grad() torch.save({'P': prompts.state_dict(), 'g': gates}, 'adapter.pt') # about 4.7 MB
Credit. LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention is by Renrui Zhang, Jiaming Han, Chris Liu, Aojun Zhou, Pan Lu, Yu Qiao, Hongsheng Li and Peng Gao, of Shanghai AI Laboratory, CUHK MMLab, UCLA and CPII, published at ICLR 2024. Read it at arXiv 2303.16199; code at OpenGVLab/LLaMA-Adapter. Every number here is from that paper unless a device says it is a toy. This lesson is a reading for MIT MMAI (Spring 2026), re-taught in our own words.
Now press Present or Teach and explain, from memory, why a plain prefix hurts a frozen model, what the gate does at g = 0, and why the gate opens before the prompts learn. Then go back to the gate at the top, drag it from zero, and say which chapter explains each thing you see.