Train a model on 757 tasks written as plain instructions and it can follow a new instruction it never saw. The 11B version beats 175B InstructGPT by 9.9 ROUGE-L points.
Learn how a crowd wrote 1,616 NLP tasks as instructions, how a T5 model was taught to read them, and what the experiments say about why it works.
Choose what the model gets to read, from only a task name up to a full definition plus two examples. Watch the score climb past a model about 58 times bigger. Then we build, piece by piece, the benchmark, the training, the scoring and the experiments.
You need a rough idea of what a language model does (it reads text and writes text) and the word fine-tuning (training a model further on new examples). We build the rest from zero.
A reading for Lecture 8, Large Multimodal Models, of MIT MMAI, Multimodal AI (Spring 2026). Paper: Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks, by Yizhong Wang, Swaroop Mishra and co-authors from AI2, UW, ASU and other labs, arXiv 2204.07705.
The 3B Tk-Instruct score is the paper’s Table 4 row for a model trained on definition plus two positive examples, tested on 119 unseen English tasks with each reading shown. Copy baseline, T0 and InstructGPT are Table 3 (InstructGPT saw 20 instances per task). The task is the small-talk example of Figure 1. Different test sets, so read the gap as a guide.
Chapter 0
See why a model that reads a written task description is worth far more than a model trained once per task
Imagine you hire a new assistant. On day one you hand over a note: “Read each message from a price negotiation. Write Yes if the speaker is making small talk, such as wishing the other side a nice trip. Otherwise write No.” The assistant has never seen this job. Still, after reading the note and two sample messages, they get started.
Language models of the early 2020s could not do this. The usual recipe was one model per job. You collected thousands of labelled examples of small talk, then ran fine-tuning: you trained a model further on those examples until it did that one job well. Next week a new job arrived, and you started again. A model that detects small talk knew nothing about rewriting questions or fixing grammar.
This lesson teaches the paper that asks a bigger question. Can one model learn to read the note? If it can, then a new job costs you one paragraph of writing, not one month of labelling.
A task is a rule that turns input text into output text. “Translate French to English” is a task. So is “answer Yes or No: is this small talk?” An instance is one input with its correct output. A task is the rule; an instance is one case of it.
An instruction is the written note that explains a task to a reader who has never met it. In this paper an instruction has a definition and a few worked examples. It is declarative: it says what the job is, not how to do it step by step.
Cross-task generalization means doing well on tasks that were never part of training, using only the instruction. This is the paper’s whole target. A model that scores well on tasks it practised may have learned only those tasks, and nothing about reading.
Other groups had already shown that training on many instructions helps. OpenAI’s InstructGPT, Google’s FLAN and the BigScience T0 model all pointed that way. But the authors name two gaps. First, the data behind the strongest models was not public, so nobody could tell which design choice mattered. Second, models as large as InstructGPT (175 billion parameters, the numbers a model adjusts while learning) were too big for most labs to retrain.
So the paper builds both missing pieces in the open. The first is Super-NaturalInstructions, nicknamed Sup-NatInst: a meta-dataset, which means a dataset made of datasets. It holds 1,616 tasks, 76 task types, 55 languages and about 5 million instances. 88 contributors wrote it. The second is Tk-Instruct, a model trained on it, which we meet in Chapter 4.
The headline result is the reason this paper is on the reading list. An 11-billion-parameter Tk-Instruct beats the 175-billion-parameter InstructGPT by 9.9 ROUGE-L points on 119 unseen English tasks. (ROUGE-L is a score for how close two texts are; Chapter 5 builds it.) A multilingual version beats it by 13.3 points on 35 non-English tasks. Try the hero again. Only one thing changes between the weak bar and the strong bar: the parts of the instruction the model may read.
Before the machinery, feel the economics. The device below uses the benchmark’s own averages. The average task has 3,106 labelled instances. The average instruction has a 56.6-word definition plus 2.8 positive and 2.4 negative examples. Drag the number of new tasks and compare what each route asks of a person.
Each bar is on a log scale, so every gridline is ten times more. Drag the slider to add tasks and watch the gap between the labelled-data bar and the instruction bars.
The averages come from Table 2 of the paper. Real tasks vary a lot, and a fine-tuned model may need fewer than 3,106 instances. The device shows the shape of the cost, not a quote.
Notice what the instruction route does not promise. It does not say the model will match a model fine-tuned on thousands of labelled instances. The paper measures that gap too, and it is large: an estimated upper bound for supervised training is 74.3 against Tk-Instruct’s 62.0 (Chapter 6). The claim is narrower and still valuable: a decent first answer from nothing but a note.
The paper has a clean order: define the data, build it, define the test, train, score, report, probe. This lesson follows it.
This paper is a reading for Lecture 8 on large multimodal models. Its link is the idea of one model steered by written instructions. The authors even list vision and speech as the natural next step in their limitations.
Chapter 1
Build the one record every task in the benchmark shares, and watch it turn into the text a model reads
Think of a recipe card. It has a title, a short description, a worked example and a warning like “do not use salted butter”. Every card in the box has the same fields, so a cook can pick up any card and know where to look. The benchmark does the same for NLP tasks. All 1,616 of them use one schema, a fixed list of fields, so one model can learn to read every task the same way.
The schema has three parts. The definition says in plain language how an input text becomes an output text. The positive examples show an input, its correct output and a short explanation of why it is correct. The negative examples show an input with a wrong output and an explanation of why it is wrong. A fourth field, the task ID, names the task, for instance task828_copa_cause_effect_classification.
Figure 1 of the paper shows a task from a negotiation dataset. The definition asks for “Yes” if the speaker uses small talk, a cooperative strategy for building rapport. The positive example is a line wishing the other side “a wonderful camping trip”, with output Yes. The negative example is a line that only moves the negotiation forward, shown with the wrong output Yes and the note that the right answer is No.
The instruction is the same for the whole task. The questions change. An instance is one input plus a list of acceptable outputs, because some tasks have several right answers. The paper puts every task in this unified format and caps each task at 6,500 instances so that big tasks do not drown small ones.
The authors also simplified an older schema from their earlier benchmark, Natural Instructions, which had 61 tasks. They merged three old fields (Definition, Things to avoid, Emphasis and caution) into one definition, and dropped the Title and Prompt fields because the definition already covered them. Fewer fields made it easier for volunteers to write tasks.
A model reads a string of text, not a record. So the paper defines an encoding: a fixed template that turns an instruction and one instance into a single text. Write the instruction of task t as It and an input as x. The template is a function called enc. The model M reads the encoded text and writes the output y.
The template is plain. It prints “Definition:” and the definition. For each positive example it prints “Positive Example 1”, then the input, the output and, if used, the explanation. It does the same for negative examples. Last it prints “Now complete the following example”, the new input, and “output:” with nothing after it. The model must fill in that blank.
By default the paper uses the combination that works best: the definition and two positive examples, with no negative examples and no explanations. Chapter 8 shows why. Build the prompt yourself below, with a real task from the paper’s Table 7. Switch parts on and off and watch the prompt grow.
The bar shows how many words each part adds. The text below is what the model would read before it writes the answer. Press the parts to include them.
Task 828, cause and effect, from Table 7 (COPA). The paper uses two positive examples by default; Table 7 prints one, so the device shows one. Word counts are of this text, not model tokens.
Symptom: you test a model on a new task and it writes a whole sentence where the answer should be one word. Measurement: the paper trains a model on definition plus two positive examples, then tests it with parts removed (Table 4). With both parts the score is 54.3. With only the definition it is 42.1. With only two examples it is 49.0. With only the task ID, a short name, it falls to 12.4.
The fix is in the schema. The definition fixes the format (“Label the instances as cause or effect”) and the examples fix the feel. Keep both, and the model is much harder to derail.
Chapter 2
Follow one volunteer’s task through three checks, and see how the finished benchmark compares with earlier ones
A graduate student in a class decides to add a task. She picks a public dataset of reading questions, writes a definition in her own words, copies in two correct examples and one wrong one, and uploads a file. Eighty-seven other volunteers are doing the same thing this month. Nobody told them how to phrase anything beyond the schema of Chapter 1. How does this pile become a benchmark you can trust?
The authors ran the project as a community effort on GitHub. A pull request is a proposed change to a shared code repository: a contributor adds a file, and maintainers decide whether to merge it. Each task arrived as a JSON file in a pull request. Contributors were NLP practitioners who answered a public call, plus students of an Arizona State University course (CSE 576) who added tasks as a class project. In all, 88 people contributed.
Contributors could source a task in three ways. They could take an existing public NLP dataset, and when the dataset came with the instructions its own annotators used, reuse them. They could use intermediate annotations from a crowdsourcing project, such as the step where workers paraphrase a question or rate its quality. Or they could invent a synthetic task that an average person understands in a few sentences, such as comparing two numbers or finding the longest palindrome inside a string.
Reusing the original annotator instructions is a smart trick. Those instructions were already tested on real people who had to produce the labels. So they were known to be enough for a human to solve the task.
Quality control had three phases. Think of them as a gate, a second reader and a stranger.
Why the label-imbalance rule? Suppose a task has 100 instances and 95 of them are “No”. A model that always writes “No” scores 95%, and nobody learns anything. The gate refuses such tasks. For the 154 test tasks the authors went further and balanced the labels, which is why a baseline that copies a random demo output behaves like a coin flip on classification.
Every task is also tagged on three axes: task type (the shape of the mapping, such as question answering or classification), language and domain (politics, medicine, dialogue and so on). Table 2 gives the totals. Work one out by hand. There are 1,616 tasks with 3,106.0 instances on average, so 1,616 × 3,106 = 5,019,296, which the paper rounds to 5 million. Of the tasks, 576 are not English, so 1,616 − 576 = 1,040 are English and 576 / 1,616 = 35.6% are not.
1,616 tasks · 76 task types · 55 languages · 33 domains · 56.6-word definitions · 2.8 positive + 2.4 negative examples
Table 1 lines the benchmark up against its neighbours. Use the device to flip between three columns of that table. Notice how the older Natural Instructions wrote far longer definitions (134.4 words) for far fewer tasks (61), and how FLAN’s definitions average only 8.2 words.
Pick a column of the paper’s Table 1. A grey n/a means the paper marks that value as unknown or not applicable.
All values are the paper’s Table 1. Task-type counts follow each dataset’s own labels, which the paper says are not directly comparable (Figure 2). PromptSource reports 13 types only for T0 training. InstructGPT’s data is not public.
The picture the table paints is a trade. Earlier collections were either small and long-winded, or large and terse. Sup-NatInst is the only row that has long definitions, negative examples, non-English tasks, public data and over a thousand tasks together.
The limitations section is blunt. The benchmark leans toward English. Outputs skew short, because short outputs are what existing datasets mostly have. That makes long-form writing under-represented. Keep both in mind when Chapter 6 reports scores: they are scores on a broad but skewed sample of NLP.
Chapter 3
Split the tasks into practice and exam, then learn the leak rule that stops the exam from borrowing the practice
A teacher hands out a workbook of practice problems. On the exam, every problem is one of the workbook problems with the numbers changed. Students score 95%. Did they learn the subject, or did they learn the workbook? You cannot tell. An honest exam uses a chapter the workbook never covered.
The same trap waits for a model that follows instructions. If the model trains on a small-talk detector and is tested on a different small-talk detector from the same dataset, a high score may only mean it remembered the dataset. So the paper builds the exam around the word unseen, and defines it with care.
Each task t has an instruction It and a set of instances with inputs Xt and outputs Yt. The model must produce y from x and It for every pair in the task. A task is unseen when none of its instances were used to train the model. The only signal about the task at test time is the instruction in the prompt.
The authors did not pick test tasks at random. They hand-picked 12 categories that cover words, sentences and documents, and both classification and generation. The categories are Textual Entailment, Cause Effect Classification, Coreference Resolution, Dialogue Act Recognition, Answerability Classification, Word Analogy, Overlap Extraction, Keyword Tagging, Question Rewriting, Title Generation, Data to Text and Grammar Error Correction. Together they hold 154 tasks.
The 154 tasks form two tracks. The English track has 119 tasks. The cross-lingual track has 35 tasks in other languages, and to the authors’ knowledge it is the first study of generalizing across both tasks and languages. For speed, the authors sample at most 100 instances per test task. A check by hand: 154 × 100 = 15,400, and the paper reports 15,310 test instances, so 90 instances are missing because a few tasks have fewer than 100.
Holding out categories is not enough. One public dataset often feeds several tasks. The OHSUMED medical dataset yields a keyword tagging task and a title generation task. WinoGrande yields a cause and effect task and a question rewriting task. If one is a test task and the other trains the model, the model has seen the text, its topics and its quirks.
The leak rule (a footnote in the paper) fixes this: drop from the training set any task sourced from the same dataset as any test task. The effect is large. After the 154 test tasks are removed, 1,616 − 154 = 1,462 tasks remain. The English track trains on only 757 of them, so the rule removes 1,462 − 757 = 705 tasks. The cross-lingual track trains on 1,271, removing 191.
Try it. The device below uses a ten-task pool: six entries borrow real task names from the paper’s tables, and four are our own stand-ins for training-only tasks built from the same datasets. Choose which categories to hold out, then switch the leak rule on and off.
Press categories to make them test tasks (blue). Green tasks train the model. Grey tasks are dropped because they share a dataset with a test task. Toggle the leak rule to compare.
Illustrative pool. The OHSUMED, ROCStories, WinoGrande, COPA and JFLEG task names come from Tables 5, 7 and 8; the four training-only entries are our stand-ins. The real split uses 154 test tasks and 757 English training tasks.
Look at what happens with Title Generation held out and the rule on: the OHSUMED summarization stand-in disappears from training, because a medical-abstract task is now in the exam. That is the rule doing its job. The price is a smaller training set, and the paper accepts that price.
Chapter 4
Trace one training example from an encoded prompt to a loss number, and see the settings that make a run take four hours
Back to the new assistant from Chapter 0. Over a few weeks you give them 757 different notes, each with a stack of solved cases, and you correct their answers. They get better at reading notes in general. On day 758 you give them a note they have never seen. That is exactly how Tk-Instruct is made. It is a model fine-tuned on the training tasks, then tested on the held-out ones.
The paper starts from T5, a model from Google built for a single trick: every job is text in, text out. T5 is an encoder-decoder network, which has two halves. The encoder reads the whole input at once and turns it into a list of number vectors, one per token. A token is a word piece, such as “cause” or “un”. The decoder then writes the output one token at a time. At each step it looks at the encoder’s vectors and at the tokens it has already written.
That fits the instruction setup perfectly. Chapter 1 defined enc(It, x), one text holding the definition, the examples and the new input. The encoder reads all of it. The decoder writes y. The same weights serve every task, because the task lives inside the text, not inside the network.
Model sizes matter later, so fix them now. The main model is built from the T5.1.1 XXL checkpoint with 11 billion parameters. For analysis the authors also train smaller T5 sizes, including the 3-billion-parameter one that most analysis uses. The multilingual variant, mTk-Instruct, starts from mT5 and has 13 billion parameters.
Each training step follows the same five moves.
The penalty is the cross-entropy loss. For one true token, it is the negative natural log of the probability the model gave that token. If the model gave the right token probability 1, the loss is 0. If it gave 0.01, the loss is large (4.6). Summed over output positions and averaged over examples, the paper’s training objective is this:
The paper states this setup in words and does not print the formula; this is the standard loss for a text-to-text model like T5. Work one example. The true output is the single word “Yes” followed by the end-of-text token. Say the model gives “Yes” probability 0.62 at position 1, and gives the end token 0.97 at position 2. The losses are −ln 0.62 = 0.478 and −ln 0.97 = 0.030. Their mean is (0.478 + 0.030) / 2 = 0.254. Drag the confidence in the device to see the loss fall as the right token gains probability.
The encoder box holds the prompt. The decoder writes “Yes” then the end token. Press Step to move through the positions, and drag the slider to change how sure the model is of the first word.
The probabilities are illustrative, chosen for a one-word answer. The other two candidates share the remaining probability in the ratio 3 to 1. The loss formula is the standard one for T5 and not printed in the paper.
Follow the tensors for the smaller models in Appendix D, where the batch size is 16, the input is cut at 1,024 tokens and the output at 128. The token ids go in as shape [16, 1024]. The encoder returns vectors of shape [16, 1024, d], where d is the model width. The decoder produces scores over the vocabulary of about 32,000 word pieces at each output position, shape [16, 128, V]. Padding is masked so empty positions add no loss.
| Setting | 11B Tk-Instruct | Smaller T5 sizes |
|---|---|---|
| Framework | Google T5 library, TPUs | Hugging Face transformers, GPUs |
| Starting point | T5.1.1 XXL, pretrained only on C4 | T5 checkpoints from the Hub |
| Hardware | Google V3-256 TPUs | 8 A100 GPUs, DeepSpeed, bfloat16 |
| Batch | 1,048,576 tokens (1,024 examples) | 16 examples |
| Learning rate | 1e-5, constant | 1e-5, constant |
| Length | 1,000 steps | 2 epochs, input at most 1,024, output at most 128 |
| One run takes | 4 hours | 6 hours |
Two numbers are worth checking by hand. 1,048,576 tokens over 1,024 examples is exactly 1,024 tokens per example, so each example fills the length limit. And 1,000 steps of 1,024 examples is about 1.02 million examples in total. A learning rate of 1e-5 is small, which means the weights move only a little: this is a gentle fine-tune, not a retraining.
One choice is a useful failure story. Plain T5 was pretrained to fill in gaps, so it cannot continue text well. The T5-LM baseline (a version further trained as an ordinary language model) scores only 30.2 ROUGE-L. The authors also tried starting Tk-Instruct from the T5-LM checkpoint, and the final result was worse than starting from T5.1.1 XXL, so they dropped it. The footnote gives no number, so take it as a report, not a measurement.
Chapter 5
Derive the score the whole paper rests on, work it by hand, and learn the three cases where it misleads
A story ends and the task is to write a title. The reference title is “Helen finds her dog”. The model writes “Helen finds her lost dog”. A strict grader who needs an exact match gives zero. A person gives nearly full marks. With 154 test tasks and open-ended answers, the paper needs a grader that sits between those two.
Many earlier papers dodged this. They gave the model a list of options and checked whether it scored the right option highest, which is called option ranking. This paper refuses. The authors argue that letting the model write freely, with no task-specific assumptions, is a more realistic test of generalizing to new tasks. The price is that scoring needs a soft metric.
A subsequence of a sentence is a set of its words kept in the original order, with gaps allowed. “Helen her dog” is a subsequence of “Helen finds her dog”. The longest common subsequence, or LCS, of two texts is the longest run of words that appears in both, in the same order. Order matters. Adjacency does not.
There is a neat way to find it. Make a grid with the reference words down the side and the prediction words across the top. Fill cell (i, j) with the LCS length of the first i reference words and the first j prediction words. If word i of the reference equals word j of the prediction, the cell is the diagonal cell plus 1. If not, it is the larger of the cell above and the cell to the left. The bottom-right cell is the answer. Tracing back through the cells that added 1 marks which words matched.
ROUGE-L, from Lin (2004), turns the LCS length into a number between 0 and 1 with two ratios. Precision asks: of the words the model wrote, what share are in the LCS? Recall asks: of the words in the reference, what share did the model cover? The final score is their harmonic mean, the F-score, which is high only when both are high.
The paper cites the metric and does not print this formula, so we give the standard F-measure form. Work two cases fully. First, reference “a dog ran home” (4 words) and prediction “the dog ran away from home” (6 words). The LCS is “dog ran home”, length 3. So P = 3/6 = 0.50, R = 3/4 = 0.75, and F = 2 × 0.50 × 0.75 / (0.50 + 0.75) = 0.75 / 1.25 = 0.60. Second, the title case: LCS is 4, prediction has 5 words, reference has 4. P = 0.80, R = 1.00, F = 1.6 / 1.8 = 0.889.
Now play with it. Type any two texts, or use a preset. The words that make up the LCS light up in the prediction.
Edit either box, or press a preset. Green words are in the longest common subsequence. Case and punctuation are ignored.
Our implementation splits on spaces after lowercasing and removing punctuation, and uses the F-measure of Lin (2004). Real ROUGE packages may also stem words, so third decimals can differ.
Symptom: a metric that rewards word overlap could rank systems in a silly order. Measurement: the authors checked in two ways. For classification-like task types, ROUGE-L tracks accuracy with a Pearson correlation of 0.970 (Figure 9). And across five systems, crowdworkers judged whether each answer was at least as good as the ground truth. The ROUGE-L scores (14.2, 32.3, 52.1, 54.3, 62.0) and the human rates (15.2, 40.0, 64.6, 64.1, 76.9) from Figure 3 give a Pearson coefficient of 0.998. We recomputed it from those ten numbers and agree.
Two cautions keep that number honest. Five systems is a small sample. And the two metrics disagree about one pair: ROUGE-L puts Tk-Instruct 3B above InstructGPT, while the crowd rated InstructGPT slightly higher (64.6 to 64.1).
The fix for the metric’s blind spots is to know them. Overlap rewards copying: a model that echoes its input scores well on rewriting and grammar correction, because the right answer is nearly the input. The paper says this itself, and that is why Copying Instance Input is a named baseline. Overlap also ignores meaning: a flipped sentence reuses the same words in the wrong order and still scores 0.60. And the score for a one-word label is all or nothing.
Chapter 6
Read Table 3 row by row, learn what each baseline is there to prove, and measure how far the best model sits from the ceiling
A result means little until you know what it was compared against. Imagine a runner finishes a race in 62 seconds. Is that fast? It depends on the other runners, on a person who simply walks the course, and on the world record. Table 3 is built the same way: it has walkers, rivals and a record. Reading it well is the skill this chapter teaches.
A heuristic baseline is a dumb rule that needs no learning. It tells you how much of the score the data hands out for free. The paper uses two. Copying Demo Output writes the output of a random example from the instruction. Because the test labels are balanced, this works like a coin flip on classification. Copying Instance Input writes the input back. It does well on rewriting and grammar fixing, where the answer is nearly the input.
Look at the cross-lingual column. Copying a demo output scores 50.3. That is high, and it is the number to remember: on many of those tasks the output is a short label, so a lucky copy earns half marks. Any model must clear that bar before its score means anything. GPT-3, with no instruction tuning, scores 51.3, barely above the walker.
T5-LM (11B) is the plain pretrained counterpart of Tk-Instruct: same size, same family, no instruction data. GPT-3 (175B) is a very large pretrained model that can copy a pattern from examples placed in its prompt. T0 (11B) is T5 fine-tuned on a collection of prompts from PromptSource. InstructGPT (175B) is GPT-3 further trained with human feedback. The authors queried the last two GPT models through the OpenAI API with temperature 0, a limit of 128 output tokens, and only 20 instances per test task because of cost. All models received the same encoding: the definition and two positive examples.
| Method | English (119 tasks) | Cross-lingual (35 tasks) |
|---|---|---|
| Copying instance input | 14.2 | 5.4 |
| Copying demo output | 28.5 | 50.3 |
| T5-LM (11B) | 30.2 | n/a |
| GPT-3 (175B) | 45.0 | 51.3 |
| T0 (11B) | 32.3 | n/a |
| InstructGPT (175B) | 52.1 | 52.8 |
| Tk-Instruct (11B, ours) | 62.0 | n/a |
| mTk-Instruct (13B, ours) | 57.1 | 66.1 |
| Supervised training (estimated upper bound) | 74.3 | 94.0 |
Read the claims off it. Instruction tuning helps: Tk-Instruct (62.0) beats T5-LM (30.2) at the same size, and InstructGPT (52.1) beats GPT-3 (45.0). The margin over the strongest rival is 62.0 − 52.1 = 9.9 points in English, and 66.1 − 52.8 = 13.3 in the cross-lingual track, the two numbers in the abstract. A model with 11 billion parameters did it against one with 175 billion, which is 175 / 11 ≈ 16 times bigger.
One surprise needs an explanation. T0 is built the same way, yet scores 32.3, only 2.1 above T5-LM. The authors suspect its prompts were styled very differently from the long, definition-style instructions here, so it did not read them well. The lesson: instruction tuning is not one thing. The style of the instructions in training must match the style at test time.
The last row is an oracle. The authors fine-tune a T5-11B on 1,000 labelled training instances for each of the test tasks themselves, which is cheating on purpose, to learn how far generalization sits from supervision. Express each score as a share of that ceiling. In English, Tk-Instruct reaches 62.0 / 74.3 = 83% and InstructGPT 52.1 / 74.3 = 70%. In the cross-lingual track, mTk-Instruct reaches 66.1 / 94.0 = 70%. The authors call this “a sizable gap”, and Figure 4 shows it on every one of the 12 categories.
Pick a view. The dashed line is the ceiling from supervised training, where the paper gives one. Systems with no value in the paper are left out.
English and cross-lingual bars are ROUGE-L from Table 3. The human view is Figure 3: the share of answers crowdworkers rated at least as good as the ground truth, on the paper’s five systems, with no ceiling shown there.
The human study agrees with the metric. Crowdworkers saw a model answer and the ground truth and said which they preferred, with ties allowed. Tk-Instruct 11B was rated at least as good as the ground truth for 76.9% of instances, which the abstract rounds to 77%. A ceiling of 100% would mean the model was never worse.
Chapter 7
Turn three dials one at a time, find which one saturates, and see how many tasks it takes to match a bigger model
You have a fixed budget to teach a student to follow notes. You can hire more teachers, so the student meets more different jobs. You can ask each teacher for more practice problems per job. Or you can pick a bigger-brained student. Which spend works best? The paper answers with three clean experiments, one per dial. To save compute, this analysis uses the 3-billion-parameter T5 and the English track, except where the dial is model size.
The authors fine-tune on random subsets of the training tasks and test on the same unseen tasks. The score rises steadily with each doubling. The paper calls this log-linear. The score is a straight line in the logarithm of the number of tasks. So each doubling of tasks gains about the same number of points. Earlier work saw this at a much smaller scale. This paper shows it still holds with 757 training tasks.
Work the slope out. With 128 training tasks the 3B model scores 48.4. With all 757 it scores 54.3. The task count grew 757 / 128 = 5.9 times, which is log2(5.9) = 2.56 doublings. The gain was 54.3 − 48.4 = 5.9 points. So each doubling is worth about 5.9 / 2.56 = 2.3 points. Notice that the line has no sign of flattening, which is why the authors call it a promising trend.
Here the number of tasks stays fixed and the number of training instances inside each task changes. Common wisdom says more data helps. The result is the opposite. Performance peaks at 54.7 with 64 instances per task, then wobbles at 53.7, 53.2 and 53.7 with many more. The paper says it “saturates” at 64.
The symptom is a flat or slightly falling curve after a point. The measurement is the set of scores above, from 54.7 down to 53.2. The reason the authors give is that extra instances only add training time and the risk of overfitting to the training tasks, which means learning their quirks and not how to read instructions. The fix is to spend that budget on a new task, not on a thousand more instances of an old one.
The authors start Tk-Instruct from five T5 sizes, from small to XXL (about 60 million to 11 billion parameters). Each larger size is better, again log-linearly: 40.1, 42.1, 48.0, 54.3 and 62.0. This contradicts a claim by Xu et al. (2022) that model size matters little once there are very many tasks.
Work the size gains by hand. From large (48.0) to XL (54.3) the score rises 6.3 points while the model grows about four times, from roughly 0.8 billion to 3 billion parameters. From XL to XXL (62.0) it rises 7.7 points for another growth of about 3.7 times. The gain per step does not shrink, which is why the paper calls the trend promising. The authors did not train anything larger, because their compute budget stopped them.
Now put dials one and three on the same page, as the paper does. A T5-large trained on all 757 tasks scores 48.0. A T5-3B trained on only 128 tasks scores 48.4. These are nearly equal. So diversity of tasks can stand in for size: a smaller model with many tasks catches a bigger model with few. The comparison is not free. A 757-task dataset costs a crowd months to write. But it is a one-time cost that every later model can reuse.
Pick a dial, then drag the slider to move along its curve. The read-out says what each step bought you.
Scores are from Figure 5 (T5-3B on the English track, except the size dial). The horizontal spacing of points is by eye on the figure’s log axes; only the points marked 128 tasks, 757 tasks and 64 instances are named in the text.
| Dial | Effect on unseen tasks | Evidence |
|---|---|---|
| More tasks | Rises log-linearly, no sign of flattening | 43.7 to 54.3 over the tested range (Fig. 5a) |
| More instances per task | Saturates at about 64 | Peak 54.7, then 53.2 to 53.7 (Fig. 5b) |
| Larger model | Rises log-linearly | 40.1 to 62.0, small to XXL (Fig. 5c) |
Chapter 8
Read a 10 by 10 grid of scores cell by cell and learn which parts of the prompt carry the weight, and which break when removed
A cook tastes a soup, then removes one ingredient at a time to learn what each one does. Remove the salt and the soup is flat. Remove the parsley and nobody notices. This is an ablation: take a part away and measure the loss. Chapter 1 gave the instruction several parts. Table 4 of the paper is the soup tasting for them.
An encoding here is a choice of which parts go into the prompt. The paper tries ten. Task ID is only a short name made of the dataset and category. Def is the definition. Pos(k) is k positive examples and Neg(k) is k negative ones. Expl adds the explanation to each example. So “Def + Pos(2) + Neg(2)” means a definition, two good examples and two bad ones.
The study trains a separate 3B model for each of the ten encodings (the rows) and tests each one on all ten (the columns). That gives 100 numbers. The 10 cells on the diagonal are models trained and tested the same way. The cells off the diagonal ask a harder question: if the user later writes the prompt differently from training, does the model cope?
Follow the diagonal from the weakest prompt to the strongest. A task name alone gives 21.2. A definition alone gives 45.0. One positive example alone gives 43.9. Two give 49.3 and four give 49.8. The definition together with one example gives 52.4. The definition with two examples gives 54.3. With four it gives 53.5.
Three things follow. The definition always helps: adding it to examples lifts 49.3 to 54.3. Examples help, then stop helping: going from 2 to 4 changes nothing, and the paper calls it negligible. And the task ID alone carries little: the dataset name tells the model almost nothing about the job.
Two weaker parts. The paper says negative examples “help a little bit”. In the table the two diagonal cells are both 54.3, so any gain is within rounding. Explanations hurt: with them the diagonal falls to 52.6, a loss of 1.7. The authors, citing earlier work, suspect that models this size cannot use explanations well, and leave bigger models to future work. That is why the default prompt is Def + Pos(2) with no negatives and no explanations.
Here the table shows a negative result and a positive one. The negative result is a one-sided collapse. A model trained on two examples with no definition scores 49.3 on its own format, but only 22.4 if a user gives it a definition and nothing else. A definition-only model fares better but still drops: 45.0 on its own format, and 36.4 when tested on two examples alone. Each single-source model leans on the source it was trained with.
The positive result is robustness. Take the default model, trained on Def + Pos(2). Test it with only the definition and it scores 42.1. Test it with only two examples and it scores 49.0. Test it with the definition and four examples and it scores 54.8, a hair above its diagonal 54.3. It handles almost every format except the bare task name, where it gets 12.4.
Use the grid. Pick a training encoding and a test encoding, or click a cell. The read-out compares it with the diagonal.
Rows are the encoding used in training. Columns are the encoding used at test time. Shorthand: T is task ID, D is definition, P is positive examples, N is negative, E is explanation. Click any cell.
All 100 values are the paper’s Table 4 (ROUGE-L, T5-3B, 119 English test tasks). Darker means higher. Outlined cells are the diagonal.
Put both a definition and a couple of clear examples in. Do not bother with more than a few. Skip explanations for a model of this size. And if you train a model to follow instructions, train it on mixed formats, because a model trained on one format learns to depend on it. The default Tk-Instruct recipe, definition plus examples, is also the one that stays robust, which matters because real users write prompts in many shapes.
Chapter 9
Keep the numbers, check your own setup against the paper’s lessons, read the core as code, and see where to go next
The paper makes one claim and backs it three ways. The claim: a model fine-tuned on many tasks written as plain instructions can follow the instruction for a task it never saw. The first backing is a benchmark of 1,616 community-written tasks with a strict unseen-task split. The second is Tk-Instruct, a T5 model that beats the 16-times-larger InstructGPT by 9.9 ROUGE-L points. The third is a set of experiments that say why: task diversity and model size both help, instances per task saturate, and a definition plus two examples is the best prompt.
| Item | Value | Where |
|---|---|---|
| Benchmark size | 1,616 tasks, 76 task types, 55 languages, 33 domains, about 5 million instances | Table 2 |
| Instruction | 56.6-word definition, 2.8 positive and 2.4 negative examples on average | Table 2 |
| Contributors | 88, three quality checks, 4 to 6 review rounds a task | Section 3 |
| Test split | 12 categories, 154 tasks (119 English, 35 cross-lingual), 15,310 instances | Section 5.1 |
| Training tasks | 757 English track, 1,271 cross-lingual track, after the leak rule | Footnote 6 |
| Default prompt | Definition plus 2 positive examples, no negatives, no explanations | Section 4 |
| Training run, 11B | T5.1.1 XXL, 1,000 steps, batch of 1,048,576 tokens, learning rate 1e-5, 4 hours on V3-256 TPUs | Appendix D |
| English score | Tk-Instruct 62.0 vs InstructGPT 52.1 (gap 9.9); supervised ceiling 74.3 | Table 3 |
| Cross-lingual score | mTk-Instruct 66.1 vs InstructGPT 52.8 (gap 13.3); ceiling 94.0 | Table 3 |
| Human check | 76.9% rated at least as good as the ground truth; Pearson 0.998 with ROUGE-L | Figure 3 |
| Scaling | Log-linear in tasks and in model size; saturates at 64 instances per task | Figure 5 |
| Best encoding (3B) | Def + Pos(2) at 54.3; explanations lower it to 52.6 | Table 4 |
The terms, in one line each. Instruction: a definition plus examples that describe a task. Instance: one input with its acceptable outputs. Unseen task: a task with no instances in training and no dataset shared with a training task. Encoding: the template that turns instruction and input into one text. Teacher forcing: feeding true previous tokens to the decoder while training. ROUGE-L: the F-score of the longest common subsequence. Ablation: removing a part to measure its effect.
Use these when you build or judge an instruction-following setup of your own. Tick what is true for yours. The paper’s results speak for the first five; the last two are cautions it gives.
Press each statement that is true for your setup. The bar fills with the share of the paper’s advice you follow.
A teaching aid, not a score from the paper. Each line restates a finding: Tables 3 and 4, Figure 5, footnote 6, Section 6.2 and the Limitations section.
This is the whole method in about forty lines: build the prompt, score the loss, score the answer, evaluate on unseen tasks. Fine-tuning itself is the standard T5 loop and is not repeated. The two asserts at the bottom check the worked numbers from Chapters 4 and 5.
python import math, re def encode(definition, positives, x, k=2): """enc(I_t, x): the text the encoder reads (Figure 8). Default: definition + 2 positives.""" parts = ["Definition: " + definition] for n, (inp, out) in enumerate(positives[:k], 1): parts.append(f"Positive Example {n}-\ninput: {inp}\noutput: {out}") parts.append(f"Now complete the following example-\ninput: {x}\noutput:") return "\n".join(parts) def loss(prob_of_true_tokens): """Mean cross-entropy over output positions (teacher forcing): -mean(ln p(true token)).""" return -sum(math.log(p) for p in prob_of_true_tokens) / len(prob_of_true_tokens) def lcs_len(a, b): d = [[0] * (len(b) + 1) for _ in range(len(a) + 1)] for i in range(1, len(a) + 1): for j in range(1, len(b) + 1): d[i][j] = d[i-1][j-1] + 1 if a[i-1] == b[j-1] else max(d[i-1][j], d[i][j-1]) return d[-1][-1] def rouge_l(ref, pred): """F-measure of the longest common subsequence, 0 to 100.""" tok = lambda s: re.findall(r"[\w']+", s.lower()) a, b = tok(ref), tok(pred) l = lcs_len(a, b) if l == 0: return 0.0 p, r = l / len(b), l / len(a) return 100 * 2 * p * r / (p + r) def evaluate(model, tasks, per_task=100): """Score a model on unseen tasks: average the per-task mean ROUGE-L.""" scores = [] for t in tasks: # t has .definition, .positives, .instances s = [rouge_l(y, model(encode(t.definition, t.positives, x))) for x, y in t.instances[:per_task]] scores.append(sum(s) / len(s)) return sum(scores) / len(scores) # checks against the lesson's worked numbers assert abs(rouge_l("a dog ran home", "the dog ran away from home") - 60.0) < 1e-9 assert abs(loss([0.62, 0.97]) - 0.254) < 1e-3
Read it against the lesson. The function encode is the Figure 8 template of Chapter 1, with two positive examples. The function loss is the mean of −ln p from Chapter 4. The function rouge_l is Chapter 5. The function evaluate follows the rule of Chapter 3: only tasks the model never trained on, at most 100 instances each. Chapters 6 to 8 report what you get from running it.
The paper belongs to the wave of work that turned raw language models into instruction followers in 2021 and 2022. Its distinct offering is openness: public tasks, public instructions, and a recipe small groups can rerun. For MMAI it is a reading for Lecture 8, because the same idea, a model steered by written instructions, is how large multimodal models are taught to follow a request about an image. The authors list vision and speech instruction following as the natural extension.
Limits to keep in mind. The data leans toward English and toward short outputs. ROUGE-L can overrate copying. InstructGPT’s training data is hidden, so overlap with the tests is unknown. The authors also trained only models they could reach, so larger ones are untested.
Credit. Super-NaturalInstructions was written by Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi and about 35 more co-authors, with Chitta Baral, Yejin Choi, Noah A. Smith, Hannaneh Hajishirzi and Daniel Khashabi leading, from the Allen Institute for AI, the University of Washington, Arizona State University and other institutions. Read the original: arXiv 2204.07705. Data, models and leaderboard: instructions.apps.allenai.org.
Now press Teach and explain the paper from memory: what an instruction contains, what unseen means and why the leak rule exists, how T5 is trained on the encoded prompt, how ROUGE-L is computed, and which of the three scaling dials saturates. Then go back to the instrument at the top and choose what the model reads once more.