LLaMA-Adapter

LLaMA-Adapter teaches a frozen 7B language model to follow instructions by training only 1.2 million new numbers, and a gate that starts at zero keeps the model’s old skills intact.

Efficient fine-tuning of language models with zero-initialized attention: how a tiny add-on learns a new job without breaking what the big model already knows.

Picture a song that already sounds right, and a new voice you want to add without spoiling it. The gate is the volume knob. Drag the gate from 0 up. Watch the right panel start exactly on the original answer, while the left panel, a plain add-on, is knocked off it before any training. Then we build, piece by piece, the prompts, the gate, the training and the picture version.

You need a feel for what a vector is (a short list of numbers, here an arrow on a page) and the idea that a model is a stack of layers. We build attention and everything else from zero.

Reading for MIT MMAI: Multimodal AI (Spring 2026, Paul Liang), paired with Lecture 8, Large Multimodal Models. Paper: Renrui Zhang, Jiaming Han, Chris Liu, Aojun Zhou, Pan Lu, Yu Qiao, Hongsheng Li and Peng Gao, ICLR 2024, arXiv 2303.16199. Re-taught here in our own words and devices.

Open the gate

0.00

Toy attention head: 4 word tokens and 3 adaption-prompt tokens, each with a 2-number value, so one output is an arrow on a page. The scores and values of the prompts are random draws made in your browser; the maths is the paper’s Equation 7: plain mixes everything in one softmax, the gate multiplies the prompts’ own softmax by tanh(g) and leaves the word softmax alone.

Chapter 0

Why Fine-tuning a Giant Is a Problem

See what it costs to retrain 7 billion numbers for one new skill, and what the paper does instead

Picture a very well-read assistant who has read most of the internet. Ask it “Tell me about alpacas” and it may not answer. It may continue your sentence, as if the question were the start of a web page. It knows a lot. It has not learned the job of answering.

That is the situation with LLaMA, a family of open language models from Meta. The 7B version has about seven billion parameters. A parameter is one learned number inside the model, like one tiny knob on a huge mixing board. Training sets all the knobs. After training, the model predicts the next word well, but it does not follow orders.

Teaching the job: instruction tuning

To teach the job, people show the model many pairs of an instruction (“Translate this into Spanish”) and a good response. Training on such pairs is called instruction tuning. It is one kind of fine-tuning: more training, on a small set of new examples, that starts from a model already trained.

The paper builds on Stanford Alpaca. The Alpaca team started with 175 instruction and response pairs written by people. They asked GPT-3.5 to invent many more in the same style, a method called self-instruct, and reached 52,000 pairs. Then they fine-tuned all 7 billion parameters of LLaMA on them. The result behaved much like GPT-3.5.

Alpaca works. The paper’s complaint is the price. Full fine-tuning is, in the authors’ words, time-consuming, computation-intensive, and cumbersome to move to other uses.

The price in numbers

Move the selector below. It shows Table 1 of the paper: three ways to turn LLaMA 7B into an instruction follower, trained on the same 52,000 pairs on 8 A100 graphics chips.

Price three ways to teach

Each bar is one method, drawn against Alpaca as 100%. Choose what to compare. The thin yellow sliver is the whole add-on.

Values are Table 1 of the paper: tuned parameters 7B, 4.2M and 1.2M; storage 13G, 16.8M and 4.7M; training time 3 h, 1.5 h and 1 h on 8 A100 GPUs. The text of Section 4.1 says 4.9M for the adapter’s storage, the table says 4.7M; we draw the table. Bar lengths are linear, so the add-on is a sliver by design.

Look at the first view. Alpaca changes 7 billion numbers. LLaMA-Adapter changes 1.2 million. That is 0.017% of the work: about one number in 5,800. The authors report one hour of training instead of three.

Now the quiet cost, storage. Say you want ten different specialists: a lawyer, a coder, a tutor, and seven more. With full fine-tuning each specialist is a full copy of the model, 13 GB each, so 130 GB. With an add-on you keep one 13 GB model and ten files of about 4.7 MB. That is 13 GB plus 47 MB. The paper calls this plug with expertise: swap the small file, change the expert.

The idea of the paper

The family of tricks for doing this is called parameter-efficient fine-tuning (PEFT). You freeze the big model, which means its parameters are never changed, and you train only a small new piece. This paper adds one such piece, with one clever detail.

The piece is a short list of learnable adaption prompts, new tokens placed inside the top layers of the model. The detail is zero-initialized attention: a gate that starts shut, so on day one the model still behaves like the original LLaMA, and the new knowledge flows in only as the gate opens. You already saw that detail in the instrument at the top of this page.

The authors list four features: only 1.2M learnable parameters, under one hour of fine-tuning, swappable adapters, and an extension that lets the same model answer questions about images. The last chapters of this lesson cover each.

The road

  1. Attention in one step: the part of a transformer the paper edits.
  2. Learnable adaption prompts: K new tokens in the top L layers, and the 1.2M count.
  3. Why random prompts hurt: the noise that a plain add-on causes.
  4. Zero-initialized attention: the split softmax and the gate.
  5. A whole layer, with shapes: Equations 1 to 8 as tensors.
  6. Training only the add-on: what is frozen, what learns, why the gate opens first.
  7. Adding eyes: one image becomes one token.
  8. What the tables show: results, ablations, limits.
  9. Field guide: numbers, terms, a checklist, the core code.
You need ten specialist versions of LLaMA 7B. Why does an adapter approach save the most disk?

Chapter 1

Attention in One Step

Follow one new word as it looks back at the words before it, and learn which parts of that step the paper can touch

You are reading “The cat sat on the” and you must guess the next word. To guess, you glance back at the earlier words. “Cat” matters a lot. “On” matters a little. You do not weigh every word equally. A language model does the same glance, many times, and the glance has a name: attention.

The paper changes only this one step inside the model. So we need to see it clearly, with every number. Nothing here is special to LLaMA-Adapter yet. It is the ordinary machinery that the adapter plugs into.

Tokens and features

A model reads tokens, which are pieces of text such as a word or a part of a word. Inside the model each token is a feature vector: a list of C numbers that describes the token in its context. For LLaMA 7B, C is 4,096. So one token is 4,096 numbers, and M tokens form a table with M rows and C columns, written M × C.

The model is a stack of N transformer layers. LLaMA 7B has N = 32. Each layer takes the table in and hands a table of the same shape to the next layer. Inside each layer, attention comes first. It lets each token read from the earlier tokens.

Query, key, value

Think of a library. You walk in with a question. Every book has a short label on its spine, and a full text inside. You compare your question with the labels, pick the books whose labels match best, and read a blend of their texts.

Attention uses the same three roles. The new token produces a query, which is its question. Every earlier token produces a key, which is its label, and a value, which is its content. Each is made by a linear projection: the token’s feature vector multiplied by a fixed table of numbers. LLaMA learned those tables in pretraining. In this paper they stay frozen.

Let the new word look back

The model is about to write the word after “The cat sat on the”. Pick the question it asks. The bars show how much it reads from each earlier word.

Illustrative: the scores below are made-up numbers chosen to look like a plausible head; a real LLaMA head has 32 of them per layer, with scores learned in pretraining. The weights are an exact softmax of the scores.

Scores, softmax, and a blend

The query is compared with each key by a dot product: multiply the two lists number by number and add up. A big result means the question matches the label. That result is a score. The scores for all earlier tokens form one row. Then the model squeezes the row into weights with the softmax, which makes every number positive and makes the row add up to 1.

S
The row of scores: one number for each token the new word can look at.
Q KT
The query times the transposed keys: one dot product per earlier token.
√C
The square root of the feature size, 64 for C = 4,096. Dot products of long vectors grow with this size, so dividing keeps the scores small enough for softmax to stay soft.
o
The output for the new word: a blend of values.
softmax(S)
The scores turned into weights that are positive and add up to 1.
V
The table of values, one row per token.

A worked row. For “Who is acting?” the five scores are 0.2, 2.2, 0.8, −0.4 and 0.0. Raise e to each: 1.22, 9.03, 2.23, 0.67 and 1.00. They add up to 14.14. Divide each by 14.14 and the weights are 0.086, 0.638, 0.157, 0.047 and 0.071. “Cat” gets 64% of the attention.

weights = [0.086, 0.638, 0.157, 0.047, 0.071],   sum = 1.000

Suppose each word’s value were the single number 4, 1, 3, 0, 2. The blend is 0.086×4 + 0.638×1 + 0.157×3 + 0.047×0 + 0.071×2 = 1.60. The new word receives a mixture that leans on “cat”. A final linear projection, Linearo, turns that blend back into a vector of size C.

Many heads, one new word at a time

LLaMA 7B does this in parallel in 32 heads. A head is one copy of the attention step with its own projections, and each works on a slice of 128 numbers (4,096 divided by 32). Different heads ask different questions. Their results are joined and mixed by Linearo.

Also remember that the model writes text one token at a time. To write token M+1, it computes the query only for the newest token and compares it with the keys and values of everything before. The paper’s equations describe exactly this step.

In one attention step, what do the softmax weights add up to, and what do they multiply?

Chapter 2

Learnable Adaption Prompts

Place K new tokens in front of the words in the top layers, and count exactly how few numbers that adds

You want to steer a student without rewriting the student’s brain. One way is to hand the student a few extra index cards before each exam question. The cards carry hints. The brain stays as it was. Only the cards are rewritten, again and again, until the hints help.

That is the plan of LLaMA-Adapter. The “cards” are adaption prompts: short lists of new, trainable feature vectors that sit in front of the real word tokens.

The background: prompt tuning

Adding trainable tokens to a frozen model is an old idea called prompt tuning. The paper cites two flavours. In one, trainable tokens are added only at the very input, next to the word embeddings (Lester et al., 2021). In the other, called prefix tuning, trainable tokens are added in every intermediate layer (Li and Liang, 2021). In both, a prefix is a block of new tokens placed before the real ones.

LLaMA-Adapter takes the second route, with two choices of its own. It uses the top layers only, and it feeds the prefix through a special gated attention (Chapter 4).

Equation 1: the prefix

Let the model have N layers. The paper picks the topmost L layers, where L is at most N, and gives each its own prompt table Pl of size K by C. K is how many prompt tokens there are. C is the feature size of the model, 4,096.

At layer l the real tokens are a table Tl with M rows: the instruction and the words written so far. The prompt is joined on top along the token direction:

[Pl ; Tl]
The K prompt rows stacked above the M word rows. The semicolon means “stack along the token direction”.
ℝ(K+M)×C
A table of real numbers with K + M rows and C columns. With K = 10 and M = 25 words, that is 35 rows of 4,096 numbers.

Every real word can now look back at K extra rows as well as at the earlier words. The rows in Pl are not words from any text. They are numbers that training will shape into whatever helps the model follow instructions.

Why the top layers?

Layers near the input handle spelling and local patterns. Layers near the output handle meaning and the plan of the answer. The authors say prompting at the last layers can better tune the language representations with higher-level semantics. Their default is N = 32 layers, K = 10, and L = 30 (Section 4.1). The ablation in Chapter 8 tests it.

Counting the new numbers

This is the arithmetic the paper’s headline rests on. Each layer has its own table of K × C numbers. There are L such layers. So the prompts hold K × C × L numbers.

10 × 4,096 × 30 = 1,228,800 numbers ≈ 1.2M

The gates of Chapter 4 add one scalar per attention head per layer: 32 × 30 = 960 more. That is too small to change the count. Stored as 4-byte numbers, 1,228,800 values take 4,915,200 bytes. That is 4.7 in units of 1,048,576 bytes, and 4.9 million bytes in decimal units. Both figures appear in the paper, in Table 1 and in the text of Section 4.1.

Try the sizes yourself. Drag K and L and watch which layers are touched and how the count moves.

Count the learnable numbers

Left: the 32 layers of LLaMA 7B, blue where frozen, pink where a prompt table is inserted. Right: the prompt rows placed in front of the word rows. Drag K and L.

10
30

Exact arithmetic: K × C × L with C = 4,096, compared with 7 billion as the paper rounds LLaMA 7B. The default K = 10, L = 30 is the paper’s setting (Section 4.1); the other settings are for you to try and were not all trained in the paper. Gates (one per head per layer) are left out of the count.

At K = 10 and L = 30 the count is 1,228,800. Cut L to 10 and it falls to 409,600. Raise K to 50 and L to 32 and it reaches 6,553,600, still under 0.1% of the model. Whatever you choose, the cost grows in a straight line with K and with L, never with the size of the frozen model.

One more fact: prompts are not shared between layers

Each layer owns a separate Pl. A hint written for layer 5 would not mean much to layer 25, because the feature vectors there describe different things. So training learns 30 independent prompt tables.

The paper uses K = 10 prompt tokens, C = 4,096 and L = 30 layers. How many prompt numbers does that give, and what is the main reason it is small?

Chapter 3

Why Random Prompts Hurt

Measure how much attention untrained prompts steal from real words, and why that damages a model that already works

A pianist plays a piece perfectly. You slip ten sheets of scribbles between the pages of the score and say, “Play as before, but learn from these.” On the first bar the pianist glances at the scribbles, and the music goes wrong. The pianist could learn, but the first notes already hurt.

The same thing happens to LLaMA when you add prompts that start as random numbers. This chapter measures the damage. The next chapter fixes it.

What a new row does to a softmax

Recall Chapter 1. A new word makes a row of scores, one per token it can read. Softmax turns the row into weights that add up to 1. LLaMA was trained with only word tokens in that row. Every head in every layer expects its weights to be split among words.

Now add K prompt rows. They enter the same row and the same softmax. The weights still add up to 1, so the prompts take their share from the words. If the prompt scores are about as large as the word scores, a simple count says how much they take. With K prompts and M + 1 word tokens, each token gets about the same weight, so the prompts get K divided by K + M + 1.

K = 10, M + 1 = 26 words:   10 / (10 + 26) = 28% of the attention goes to the prompts

For a short instruction the share is worse. With M + 1 = 6, it is 10 / 16, which is 62%. In the right head, the real words are ignored more than half the time, and what replaces them is random.

Two harms in one

First, the real words lose weight. Their weights are rescaled by one minus the prompt share, so the careful recipe LLaMA learned is diluted. Second, the weight that goes to the prompts reads their values, and a random value adds a random vector to the blend. The head’s output is pushed in a direction nobody chose.

The push does not stay in one place. There are 30 inserted layers. The output of one layer is the input of the next, so an error made in layer 3 is read by layers 4 to 32. The authors state this plainly: randomly initialized prompts bring disturbance to the word tokens at the beginning of training, which harms fine-tuning stability and effectiveness.

Measure the noise

Each setting is averaged over 400 random toy heads. The top bar splits attention between prompts (pink) and words (blue). The lower bar is how far the output moves from the original. Change K, the number of words, or how loud the prompt scores are.

10
26
1.0

Toy simulation, not from the paper: word scores are random with spread 1, values are random 2-number vectors, 400 seeded trials. Shift is the length of the output change divided by the typical length of the original output. A gate closed at g = 0 would show 0% on both bars, by construction (Chapter 4).

Start at the defaults: K = 10, 26 words, scores of size 1. About 27% of the attention goes to prompts, close to the simple count of 28%, and the output moves by about 43% of its own length. Now set the prompt score size to 0. Every prompt scores exactly 0, an average score, the share drops to about 20% and the shift to 25%. Raise the size to 3 and a few prompts get huge scores. They grab about 69% of the attention and the shift passes 160%.

Shrink K to 1 and raise the words to 60. The damage drops, because one prompt among 60 words is a small voice. This is why a tiny prompt at the very start of training might be tolerable, and ten of them at 30 layers is not.

Symptom, measurement, fix

Symptom: training begins from a model that is already worse than the one you froze. The loss starts high, falls slowly, and the model learns to answer badly.

Measurement: the paper’s random-initialized baseline reaches 40.77% accuracy on the ScienceQA validation set. Picking answers at random scores 39.83% (Table 2). So the baseline learned almost nothing useful. In the loss curves of Figure 7 it slowly approaches 0.15 and has not fully converged.

Fix: make the prompts invisible at the start. A prompt that gets zero influence cannot disturb anything. Then let the influence grow. That is the next chapter.

With K = 10 prompts and 6 word tokens, and prompt scores about as large as word scores, roughly how much attention goes to the prompts, and why is that a problem for a pretrained model?

Chapter 4

Zero-initialized Attention

Split the softmax in two and put a gate that starts closed on the prompt half

Imagine a radio with a volume knob. You want to add a new voice to a song that already sounds right. If the knob starts at the middle, the new voice is loud on the first second and spoils the song. If the knob starts at zero, the song is untouched, and you raise the new voice slowly while you listen.

This is the paper’s central trick. The new voice is the attention that goes to the adaption prompts. The knob is a learnable number called the gate, g. It starts at zero.

Step 1: scores for everything, split in two

The new word tl has its query, and it compares it with keys of the prompts, the earlier words and itself, K + M + 1 tokens in all. Equations 2 to 5 give a row of K + M + 1 scores. The paper then splits that row into two named parts (Equation 6):

Sl
All the scores of the new word at layer l, size 1 × (K + M + 1).
SlK
The K scores for the adaption prompts. The paper says these show how much information the prompt gives to the new word.
SlM+1
The M + 1 scores for the real word tokens, including the new word itself.

Step 2: two softmaxes and one gate

A plain model would run one softmax over the whole row. LLaMA-Adapter runs two, one on each part, and multiplies the prompt part by a gate (Equation 7):

Slg
The final attention weights. The superscript g means “gated”.
softmax(SlK)
The prompts’ weights, which add up to 1 among the K prompts alone.
tanh(gl)
The gate. The function tanh squeezes any number into the range −1 to 1, so a large g cannot blow up the prompts. At g = 0, tanh(0) = 0.
softmax(SlM+1)
The real words’ weights, from their own softmax, with no coefficient. They add up to 1 among the words alone.

Two design decisions sit in this line. First, the separate softmaxes. The words’ softmax never sees the prompt scores, so it equals exactly what the original LLaMA computed. The paper says this preserves the original probability distribution. Second, the gate on the prompt side only. Nothing multiplies the words’ weights, so the pretrained knowledge is not disturbed.

Step 3: the output is the old answer plus a gated nudge

Equation 8 multiplies the weights by the values and applies Linearo. Because the weights are two separate groups, the blend splits too:

blend = (words’ weights · word values) + tanh(g) × (prompts’ weights · prompt values)

The first term is the original LLaMA blend, unchanged. The second term is the prompts’ contribution, scaled by tanh(g). Close the gate and only the old answer is left. That is the whole mechanism, and the next device lets you check it with numbers.

Open the gate on numbers

Three word tokens and two prompts. Top row: one softmax over all five. Middle row: two softmaxes, prompts scaled by tanh(g). Bottom line: the blended answer. Drag g.

0.00

Worked example, illustrative numbers: word scores 1, 0, −1 with values 3, 1, 2; prompt scores 0.5, −0.5 with values 10, −4. The original answer is 2.42; the plain joint softmax gives 3.78 before training; the gated version gives 2.42 + tanh(g) × 6.24. All from the paper’s Equations 6 to 8.

Check the numbers by hand. Words: softmax of (1, 0, −1) is (0.665, 0.245, 0.090), and the blend is 0.665×3 + 0.245×1 + 0.090×2 = 2.42. Prompts: softmax of (0.5, −0.5) is (0.731, 0.269), and the blend is 0.731×10 + 0.269×(−4) = 6.24. At g = 0 the answer is 2.42 + 0 = 2.42, the original. At g = 1, tanh(1) = 0.762, so the answer is 2.42 + 0.762×6.24 = 7.17.

The plain version, a single softmax over all five scores, gives (0.429, 0.158, 0.058, 0.260, 0.096). Its words have lost a third of their weight, and it answers 3.78 at step zero. That is 56% above the original, with no training at all.

One gate per head

In practice the paper uses a separate g for every attention head, so each of the 32 heads learns its own volume. Some heads may open fast and others stay shut. The authors say this benefits the learning diversity of the multi-head mechanism.

At g = 0, what does a zero-initialized attention layer compute, and which design choice guarantees it?

Chapter 5

A Whole Layer, With Shapes

Trace Equations 1 to 8 through one inserted layer, tensor by tensor, with the real LLaMA 7B sizes

A recipe is easy to nod at and hard to cook. Here we cook it. The model has read an instruction and a few words of reply, 25 words in all. It is about to write the next one. We follow that one new token through one inserted layer, and write down the shape of every table that appears.

The settings are the paper’s: C = 4,096, K = 10 prompt rows, and M word rows already read. Take M = 25.

The seven stages

  1. Inputs. The newest token is t: 1×4096. The earlier words are T: 25×4096. This layer’s prompt is P: 10×4096, the one learned table.
  2. Stack. Join P, T and t into one table, 36×4096 (Equation 1 plus the new token).
  3. Project. Frozen LLaMA projections make the query from t alone, 1×4096, and the keys and values from the whole stack, 36×4096 each (Equations 2 to 4).
  4. Score. Compare the query with all 36 keys and divide by the scale: 1×36 (Equation 5).
  5. Split and gate. Cut the 36 scores into 10 for prompts and 26 for words. Softmax each part alone. Multiply the 10 prompt weights by tanh(g) (Equations 6 and 7).
  6. Blend. Multiply the 36 weights by the 36 value rows: 1×4096.
  7. Output. The frozen Linearo mixes the heads: 1×4096 (Equation 8). The shape equals the input, so the next piece of the layer sees nothing odd.

Step through it below. Drag the number of words and watch which shapes change and which stay the same.

Step through a layer

Press Next to light the stages in order. Pink marks the learned table and gate, blue marks frozen LLaMA. The right column is the shape at that stage.

25

Shapes follow the paper’s Equations 1 to 8 with C = 4,096 and K = 10. The paper writes the scale as the square root of C; a multi-head implementation divides by the square root of each head’s own slice, 128 numbers, and we describe the single-head reading. M is yours to set; the paper’s own M is the instruction plus the reply so far.

Two things to notice. First, only the stack and the keys and values grow when the text grows. The query stays 1×4096, and the output stays 1×4096. Second, the prompt rows travel through the frozen key and value projections of LLaMA. They are not given a new language. They must learn to be numbers that LLaMA’s own projections turn into useful keys and values.

Why only the newest token queries

The prompts never ask questions. They are rows to be read, not tokens that read. Only real tokens produce queries. So the prompts add K rows to the key and value tables and nothing else. The extra cost per layer at answer time is small: K more rows to compare, 10 against about 26 in this example.

After training, Pl is fixed. So the prompt part of the keys and values is the same for every request, a fact one could use to save work. The paper does not claim this; it follows from the equations.

What to check when you build it

Three slips are common. First, a shape slip: if the score row has 35 entries instead of 36, the new token was left out of the stack, and it cannot read itself. Second, a split slip: if you run one softmax and then cut the row in two, the word weights no longer equal the original, and the gate loses its promise. Third, a freeze slip: if any LLaMA table still has gradients switched on, you train far more than 1.2M numbers.

The core as code

Here is the same layer in Python, for one new token. It uses PyTorch. Each comment names the equation. The matrices Wq, Wk, Wv and Wo are LLaMA’s own and are never updated. Only P and gate are trained.

python
import torch, torch.nn.functional as F

def zero_init_attention(t, T, P, Wq, Wk, Wv, Wo, gate, heads=32):
    # t: (1, C) newest token   T: (M, C) earlier tokens   P: (K, C) adaption prompt
    # Wq, Wk, Wv, Wo: frozen (C, C)   gate: (heads,) learnable, starts at zero
    C = t.shape[-1]; d = C // heads; K = P.shape[0]
    X = torch.cat([P, T, t], dim=0)                  # (K+M+1, C)     Eq. 1
    q = (t @ Wq).view(1, heads, d)                   # (1, H, d)      Eq. 2
    k = (X @ Wk).view(-1, heads, d)                  # (K+M+1, H, d)  Eq. 3
    v = (X @ Wv).view(-1, heads, d)                  # (K+M+1, H, d)  Eq. 4
    s = torch.einsum('qhd,nhd->hqn', q, k) / d ** 0.5   # (H, 1, K+M+1)  Eq. 5
    sK, sW = s[..., :K], s[..., K:]                    # Eq. 6: prompts | words
    wK = F.softmax(sK, dim=-1) * torch.tanh(gate).view(heads, 1, 1)   # Eq. 7
    wW = F.softmax(sW, dim=-1)                       # Eq. 7: words, untouched
    w = torch.cat([wK, wW], dim=-1)                  # (H, 1, K+M+1)
    out = torch.einsum('hqn,nhd->qhd', w, v).reshape(1, C)
    return out @ Wo                                    # (1, C)         Eq. 8

Set gate to zeros and wK becomes all zeros. Then out is exactly what a plain LLaMA head computes, because wW is a softmax over only the words. You can test that claim with one assert, and it is a good first unit test for anyone who re-implements the paper.

With K = 10 prompt rows and M = 25 earlier words, what is the shape of the score row S, and what is the shape of the layer’s output for the new token?

Chapter 6

Training Only the Add-on

Freeze the giant, send gradients to the prompts and gates, and watch why the gate opens first

You tune a guitar by turning pegs while you listen. You do not rebuild the guitar. Training the adapter is like that: the instrument is frozen, a few pegs move, and the loss is the ear that says “closer” or “further”.

What is frozen and what learns

The whole of LLaMA is frozen: embeddings, all N = 32 layers, every Linear table, the output head. Four kinds of numbers learn, and only these:

Training uses the same 52,000 instruction and response pairs as Alpaca. The paper’s settings (Section 4.1): 8 A100 graphics chips, 5 epochs (an epoch is one pass over the data), 2 warm-up epochs (the learning rate rises slowly at first), batch size 64, learning rate 0.009 and weight decay 0.02. The paper does not spell out the loss. The standard choice for instruction tuning, and the one Alpaca uses, is to predict each next word of the response and penalise wrong guesses. We assume that here.

Frozen does not mean the error skips it

Look at the “Backward” arrow in the paper’s Figure 2. The error signal must travel from the loss at the top, back down through frozen layers, to reach the prompts in layer 3. Each frozen layer passes the signal along but does not store an update for its own weights.

The saving is in what is kept. Many optimizers keep extra numbers for every trained parameter. With 7 billion trained parameters that is a huge load. With 1.2 million it is tiny. The paper adds a point for several machines: only the gradients of 1.2M parameters must be sent between machines, not 7B. The forward and backward pass through the frozen model still costs real compute, which is why training takes one hour and not one minute.

Why the gate opens first

At the start every g is 0. Recall from Chapter 4 that the answer is: original blend + tanh(g) × prompt blend. Ask how the loss changes if each part moves a little. Two facts come from the chain rule.

  1. The gradient on g is not zero. It is the loss slope times the prompt blend times the slope of tanh at g. At g = 0 that slope is 1. So g feels the full push.
  2. The gradient on the prompts is exactly zero. The prompt blend sits behind the factor tanh(g). With tanh(0) = 0, a change in P changes nothing yet, so its gradient is 0.

So the first updates move only the gates. As soon as g leaves zero, tanh(g) is no longer zero, gradients reach the prompts, and the prompts begin to learn. This is the “gate opens first, then the prompts follow” story. The paper puts it as: zero gating first eliminates the influence of under-fitted prompts, then increases its magnitude to provide more instruction semantics. The derivation above is our reading of that sentence.

Watch it happen in a tiny version. The toy has 3 prompt rows, 2-number values and a frozen head, and its target answer differs from the frozen answer. Press Train, or drag the step.

Watch the gate open

Three curves over 60 steps of gradient descent: the loss (grey, as a share of its start), the gate tanh(g) (orange), and how hard the error pushes on the prompts (pink). Press Train to play it, or drag the step.

0

A toy, not the paper’s run: one head, 4 frozen word tokens, 3 trainable prompt rows (scores and 2-number values), one gate, squared-error loss to a made-up target, plain gradient descent at a rate of 0.1 with exact numeric gradients. Computed live in your browser. The real model has 30 layers, 32 gates per layer and a language loss.

With the first random draw, the story runs like this. At step 0 the loss equals the frozen model’s loss, and the prompt gradient is exactly 0 while the gate gradient is not. By step 10 the gate has opened about two thirds of the way and the prompt gradient is at its peak. By step 40 the loss is near 0, the prompt gradient has faded and the gate has settled. Draw new prompts and the timing shifts, but the order stays the same. The gate is a handle for the first moves, and it stays open at a level the data wants.

Symptom, measurement, fix, again

Symptom: a prefix model whose first step already hurts the loss and trains slowly. Measurement: compare the loss curve and a held-out score of the zero-init run with a random-init run, as the paper does in Figure 7 and Table 5. Fix: keep the structure of Chapter 4. Chapter 8 gives the measured numbers.

At the first training step, with every gate at zero, which gradients are nonzero?

Chapter 7

Adding Eyes: The Multi-modal Version

Turn one image into one token, add it to every prompt, and train in two stages

You show a friend a photo and ask, “What place is this?” Your friend cannot see a photo in text. But if you whisper a one-line summary of the photo into your friend’s ear, in your friend’s own language, your friend can answer. The multi-modal LLaMA-Adapter does exactly that: it whispers one summary token to the model.

The nice part is that it needs no new machinery. The summary token rides in on the prompts you already know, so it passes through the same gate.

From a picture to one token

A visual encoder is a model that turns an image into numbers. The paper uses CLIP, which was trained on pairs of images and captions (see our CLIP lesson). It stays frozen. CLIP gives multi-scale global features: several vectors Im, each of size 1 × Cm, each a summary of the whole image at a different level of detail. The paper calls the number of scales M, which clashes with the M words of the earlier chapters, so below we say “the scales”.

The scales are joined into one long vector and passed through a small trained network, the projection. The paper says it is a simple bottleneck MLP, meaning a small network that squeezes the vector and expands it again. Its output is one vector of size 1 × C, the same size as a word token (Equation 9).

Ip
The overall image token, size 1 × C. It is shaped like a word so the language model can read it.
Projection
The learned network that moves CLIP’s features into the word embedding space of LLaMA. The paper says it adds 0.6M parameters.
Concat
Join the scales end to end along the channel direction. No weights.
Im
CLIP’s global feature at scale m, size 1 × Cm.

Adding the token to every prompt

Now the key step. Copy Ip K times to make a K × C table, and add it, element by element, to the prompt table of every inserted layer (Equation 10):

Plv
The prompt that carries the picture: this layer’s learned prompt plus the image token.
Pl
The learned prompt of layer l, as before.
Repeat(Ip)
The image token copied K times, so every prompt row receives the same picture summary.

Nothing else changes. Plv goes into the zero-initialized attention of Chapter 4 in place of Pl. Its keys and values are made by the same frozen projections, and its attention is multiplied by tanh(g). So the image reaches the answer only through the gate. At g = 0 the model ignores the picture and acts like plain LLaMA. As the gates open, the picture’s influence grows.

Two stages of training

For ScienceQA, a question set with images, the authors train on its own training set in one step. They freeze CLIP and LLaMA and train the projection and the zero-init attention. That gives 1.2M (prompts and gates) plus 0.6M (projection) = 1.8M learnable parameters, as Table 2 lists.

For the open-ended tests (MME, MMBench, LVLM-eHub) they use two stages. Stage 1 aligns pictures with words: they use image and caption pairs from LAION-400M to tune the projection and the zero-init attention. Stage 2 teaches answers: they freeze the projection and tune the zero-init attention on Alpaca data mixed with LLaVA-I instruction data. Walk through the three setups below.

Freeze, or train?

Choose a setup. Each row is a part of the model. Pink rows train in that setup; blue rows are frozen; grey has no weights.

From Section 3.3 and Section 4.2. The 1.2M, 0.6M and 1.8M figures are the paper’s (Section 4.2, Table 2). Counts for the two stages follow from which parts the text says are tuned in each stage. The paper does not give a separate count for each stage.

In stage 2 only the 1.2M of prompts and gates learn. The projection that was aligned in stage 1 is frozen so that the instruction data cannot undo the alignment. That is our reading of why the authors split the stages. The paper states the split, not the reason.

How does the multi-modal LLaMA-Adapter let the model see an image, and why is it safe to start training?

Chapter 8

What the Tables Show

Read each main table and figure, compare the ablations, and see where the evidence stops

A claim is only as good as the table behind it. You now know the mechanism. This chapter reads the paper’s evidence the way a careful reviewer would: what was measured, against what, and what the number does and does not say.

The setting

Almost every result uses LLaMA 7B with N = 32 layers, K = 10 and L = 30, fine-tuned on the 52,000 Alpaca pairs for language tests (Section 4.1). The multi-modal tests use CLIP as the eye and a bottleneck MLP as the projection (Section 4.2). Choose a view below to see each main number drawn on one axis.

Read a table

Pink bars are LLaMA-Adapter. Blue bars are other methods. Red marks the random-initialized baseline. Pick a table and read the sentence under it.

Main results

Efficiency (Table 1). 1.2M tuned parameters, 4.7M storage and 1 hour on 8 A100 chips, against 7B, 13G and 3 hours for Alpaca, and 4.2M, 16.8M and 1.5 hours for Alpaca-LoRA. This is the headline and it is measured directly.

Instruction quality (Figure 4, Figure 5, Table 18). Side-by-side answers for question answering, translation and code look similar to Alpaca’s. In the GPT-4 judging benchmark, where GPT-4 compares two answers on 80 questions, LLaMA-Adapter wins more often than Alpaca and than Alpaca-LoRA. On the Open LLM benchmark (ARC, HellaSwag, MMLU, TruthfulQA) the averages are 52.2 for LLaMA-Adapter, 50.73 for Alpaca-LoRA and 49.23 for Alpaca.

Pictures (Table 2, Table 3). On ScienceQA the text-only variant reaches 78.31% and the multi-modal one 85.19%, a gain of 6.88 points from the 0.6M projection. For comparison the table lists human 88.40, GPT-4 with chain of thought 83.99 and MM-CoT 84.91. On MME perception it scores 973, against 867 for MiniGPT-4 and 503 for LLaVA. On MME cognition it scores 249, below MiniGPT-4’s 292.

Ablations: which part carries the weight

An ablation removes or changes one part to see what breaks. Two ablations matter here, both on the ScienceQA validation set.

How many layers (Table 4). With 10 layers the score is 55.95%; with 20, 73.36%; with 30, 83.85%; with all 32, 81.03%. The parameter counts are 0.97M, 1.37M, 1.79M and 1.83M. More layers help up to a point. Too many, the authors say, disturb the early encoding of the input words. They also note that with few resources, inserting into all layers is generally a good solution.

The gate itself (Table 5, Figure 7). Random initialization scores 40.77%. Zero initialization scores 83.85%, a gain of 43.08 points. Random choice on this test scores 39.83%, so the random-init run is almost at chance. In the loss curves the zero-init run falls faster at the start and converges toward zero, while the random-init run slowly approaches 0.15.

This is the strongest evidence in the paper. The same prompts, the same data, the same training time, and one change: how the attention starts.

Other models

Section 4.4 applies the same attention to ViT on VTAB-1k, RoBERTa-large on SQuAD, and CLIP on base-to-novel tests. The text reports better results than full fine-tuning on VTAB-1k and the best average on the CLIP test. On SQuAD 2.0, zero-init (87.2 F1) trails full fine-tuning (89.4) and beats the stronger prompt-tuning baseline, P-Tuning v2 (85.5). So the claim is “competitive and cheap”, not “always best”.

Where the evidence stops

Table 5 shows 40.77% for random initialization and 83.85% for zero initialization on ScienceQA validation. What can you conclude?

Chapter 9

Field Guide

Keep the numbers, the terms, a checklist and the core code on one page

Here is the whole paper in one paragraph. Full fine-tuning of a 7B model for instructions is slow and heavy. LLaMA-Adapter freezes the model and adds K = 10 learnable prompt rows in each of the top L = 30 layers, which is 1.2M new numbers. The prompts enter attention through a split softmax, and their share is scaled by tanh(g) with g started at zero, so training begins from the original model and lets instruction knowledge in as the gate opens. One hour on 8 A100 chips gives answers comparable to Alpaca. Adding one CLIP-based image token to every prompt turns the same model into a picture-and-text model.

The numbers

QuantityValueWhere
Layers N, prompts K, inserted layers L32, 10, 30Section 4.1
Feature size C4,096 (LLaMA 7B)Chapter 2 arithmetic
Learnable parameters1.2M (10 × 4,096 × 30 = 1,228,800)Abstract, Table 1
Multi-modal learnable parameters1.8M (1.2M + 0.6M projection)Table 2, Section 4.2
Storage per adapter4.7M in Table 1, 4.9M in the textTable 1, Section 4.1
Training data and schedule52K pairs, 5 epochs, 2 warm-up, batch 64, rate 0.009, decay 0.02Section 4.1
Training time1 hour on 8 A100 chips (Alpaca: 3 hours)Table 1
ScienceQA average85.19% (text only: 78.31%)Table 2
Zero-init gain40.77% to 83.85%, +43.08 pointsTable 5
Layers inserted30 layers 83.85%, all 32 layers 81.03%Table 4
MME perception, cognition973, 249Table 3

The terms

TermPlain meaningChapter
Adaption prompt PlK learned rows, one table per inserted layer, placed before the word rows2
Zero-initialized attentionAttention with two softmaxes and a gate that starts at zero on the prompt side4
Gate glA learned number per head, squeezed by tanh to the range −1 to 14
FrozenNever updated by training; gradients still pass through it6
Image token IpOne vector summarising the image, same size as a word token7
AblationA test that removes or changes one part to see its effect8

A checklist to re-implement it

  1. Freeze every LLaMA weight.
  2. Make L = 30 prompt tables Pl of shape 10 × 4,096, one for each of the top 30 layers.
  3. Make one gate per head per layer, all zeros: shape (30, 32).
  4. Replace attention in those 30 layers with the zero-init version of Chapter 5: separate softmaxes, tanh(g) on the prompt side only.
  5. Run a unit test: with the gates at zero, the logits must equal the original model’s logits.
  6. Train only prompts and gates on the 52K pairs with the Section 4.1 schedule.
  7. Save only the prompts and the gates, about 4.7 MB.
  8. For pictures: CLIP, concat, MLP, repeat K times, add to every Pl; stage 1 on captions, stage 2 on instructions with the projection frozen.
python
import torch, torch.nn as nn

K, L, C, HEADS = 10, 30, 4096, 32
llama = load_llama_7b()                                  # your loader
for p in llama.parameters(): p.requires_grad = False   # freeze all 7B

prompts = nn.ParameterList([nn.Parameter(torch.randn(K, C) * 0.02) for _ in range(L)])
gates   = nn.Parameter(torch.zeros(L, HEADS))            # zero-init: the whole trick
# the paper does not give the prompt init; the closed gate makes the start safe either way

# swap attention in the top 30 of 32 layers for zero_init_attention (Chapter 5)
opt = torch.optim.AdamW([*prompts.parameters(), gates], lr=0.009, weight_decay=0.02)
# optimizer name is our assumption; the paper gives the rate and decay only

assert sum(p.numel() for p in prompts.parameters()) == 1_228_800
for batch in alpaca_52k:                                # 5 epochs, batch 64, 2 warm-up epochs
    loss = response_next_word_loss(llama, batch)         # assumed standard LM loss
    loss.backward(); opt.step(); opt.zero_grad()
torch.save({'P': prompts.state_dict(), 'g': gates}, 'adapter.pt')   # about 4.7 MB

Where it sits in the field

Keep going

The idea to keep. To add a new skill to a model that already works, start from the answer you trust and let the new signal in through a gate that begins shut. The gate is what makes a tiny add-on safe to train.

Credit. LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention is by Renrui Zhang, Jiaming Han, Chris Liu, Aojun Zhou, Pan Lu, Yu Qiao, Hongsheng Li and Peng Gao, of Shanghai AI Laboratory, CUHK MMLab, UCLA and CPII, published at ICLR 2024. Read it at arXiv 2303.16199; code at OpenGVLab/LLaMA-Adapter. Every number here is from that paper unless a device says it is a toy. This lesson is a reading for MIT MMAI (Spring 2026), re-taught in our own words.

Now press Present or Teach and explain, from memory, why a plain prefix hurts a frozen model, what the gate does at g = 0, and why the gate opens before the prompts learn. Then go back to the gate at the top, drag it from zero, and say which chapter explains each thing you see.

LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention
Read the paper · Authors’ code · Lecture 8, Large Multimodal Models · Back to Gleams