A frontier lab does not collect training data, it manufactures it. Generating is cheap. The pipeline that checks, filters and mixes what comes out decides what the model learns.
Learn how models write their own training sets for agents and coding, how each stage keeps the bad records out, and how to build a pipeline that a whole team trusts and reuses.
Pick a filter, then drag the verifier’s strength. Watch which candidates survive and how clean the kept set is. Then we build, piece by piece, every stage: the seeds, the generator, the checks, the dedup, the mix, and the framework around them.
You need a rough idea of what a language model does (it writes the next word) and a little Python for the code. We build the rest from zero.
Built from eight primary sources, listed with links in the last chapter: Self-Instruct, WizardLM, Magpie, STaR, the Llama 3 report, APIGen, SWE-smith and The Curse of Recursion. Every toy number says it is a toy.
Illustrative batch, not a measurement: 48 correct and new, 24 correct repeats, 30 wrong answers that look fine, 18 broken records. The slider is the share of wrong answers the check catches. In this toy, a stricter check also drops by mistake up to 15% of the good records (strength squared times 15%).
Chapter 0
See why a lab that wants a coding agent cannot simply hire enough people, and what it builds instead
Picture a research team on Monday morning. They want a model that can open a code repository, find why a test fails, edit the right file and prove the fix. To teach it, they need examples: a broken repository, the steps a good engineer takes, and the passing result.
The team has a budget for people who write such examples. A careful example takes an expert maybe half an hour. Ten thousand examples is already a large project. A lab needs far more than that, for hundreds of different skills.
This lesson is about the factory that replaces that hiring plan. The factory is a synthetic data pipeline: a chain of programs in which a model writes candidate training examples, other programs check them, and only the survivors reach training. Synthetic means made by a model, not by a person.
The first reason is cost. An expert hour is expensive, and an expert is slow. A model writes a candidate in seconds, and a thousand machines write in parallel. The money per finished example falls by orders of magnitude, even after you count the checking.
The second reason is coverage. Coverage means how many of the situations you care about appear in the data at all. People write the examples that come to mind. They write fifty ways to fix an off-by-one bug and none for a flaky network test.
The third reason is the long tail. In real use, a few kinds of task are very common and a huge number of kinds are rare. Plot the kinds by how often they occur and you see a tall head and a long, thin tail. Each tail kind is rare, yet together the tails make up a large share of what users ask.
Random collection serves the head well and the tail badly. If one task kind in a hundred is the commonest, it will fill up fast. The rarest kinds will wait a very long time for even a few examples. Try it below.
Each bar is one of 100 task kinds, commonest on the left. A bar turns green when it holds 20 or more examples. Drag the collection size, then switch on Aim at the gaps.
Illustrative model, not a measurement: task kind j appears with weight proportional to 1/j (a Zipf curve over 100 kinds), and 20 examples count as covered. The expected counts are exact for that model, using the Poisson tail for the chance of reaching 20. The $3 per human example is a made-up round number for the arithmetic.
At a thousand examples, only the ten or so commonest kinds are covered. To cover the rarest kinds by luck you need about a hundred times more data than the head needs. That is the cost curve of random collection.
Now the key move. A pipeline does not have to wait for luck. It can choose which kinds to write, because a program is writing the examples. Switch on Aim at the gaps: every kind gets the same share, and all 100 are covered at 2,000 examples. A person cannot easily be told to “write a rare kind” one hundred times in a row. A generator can.
Every pipeline in this lesson has the same shape. Learn the shape first, because the chapters fill it in one stage at a time.
Notice where the value sits. Generation is the cheap, easy step. A weak generator still produces a million candidates. The hard work is steps three to six, because they decide what survives. The hero at the top of this page is that idea in miniature: the same 120 candidates, and the filter decides how clean the kept set is.
A one-off script can make a dataset once. A frontier lab needs more. Many researchers will change the generator, the checks and the mix, again and again, and they must trust the results. So the real product is not a dataset. It is a framework: stages that snap together, datasets with version numbers, a record of where each example came from, and numbers that show what each stage did. Chapter 7 builds that framework.
Also, synthetic data is not free of risk. A model that trains on its own writing can lose the rare cases it once knew. A verifier with a hole teaches the model to use the hole. A generated set can quietly contain the test questions. Chapter 8 measures each of these failures and shows the fix.
Chapter 1
Grow twenty hand-written tasks into tens of thousands by letting a model write the questions, then keep only the new ones
You have written twenty tasks by hand. Each one is a short instruction: a request that a user might send, such as “Write a Python function that reverses a string.” You need fifty thousand. Writing thirty thousand more by hand will take months.
So you ask a model to help. You show it a few of your tasks and say: write one more like these, but different. It writes one. You add it to the pile and ask again. The pile is called the pool, and the twenty hand-written tasks that start it are the seeds.
This loop is Self-Instruct. A model uses its own generations to grow a set of instructions, then the set is used to finetune a model. In the paper, applying it to vanilla GPT3 gave a 33% absolute improvement on Super-NaturalInstructions, and left a 5% absolute gap to InstructGPT-001 in human evaluation.
One round has four steps. First, sample a handful of instructions from the pool as examples, mixing seeds with earlier generated ones. Second, generate new instructions from that prompt. Third, filter the new ones. Fourth, add the survivors to the pool. Then repeat.
The filter is what keeps the loop honest. A model asked for “something new” often writes something it has already written. If you add every copy, the pool grows in size but not in range. So the filter measures how close a candidate is to its nearest neighbour in the pool, and rejects it if the score is too high.
Here is one measurement, worked by hand. A similarity score tells how many words two texts share in the same order. Take two instructions of eight words each. The pool has “Write a Python function that reverses a string.” The candidate is “Write a Python function that reverses a list.” They share seven words in order, so the score is 7 / 8 = 0.875. With a limit of 0.7 (an illustrative value), the candidate is rejected. It is a copy with one word changed.
The data the loop produces is a plain record. Keep it this way from day one, because later chapters need the extra fields.
json{
"id": "si-000412",
"instruction": "Write a Python function that merges two sorted lists.",
"input": "",
"output": "def merge(a, b): ...",
"source": "self-instruct",
"parents": ["seed-07", "si-000118"],
"gen_model": "gen-v1",
"round": 3,
"nearest_sim": 0.41
}
Now run the loop yourself. Each round proposes 30 candidates. As the pool fills up, new candidates land closer to old ones, so more of them get rejected. The threshold decides how strict you are.
Bars: new instructions accepted in each round. Line: the pool size. Lower the limit to demand more novelty and watch the growth slow.
Toy simulation, not a measurement: 20 seeds, 30 candidates per round, and a made-up rule that candidates land closer to the pool as it grows. The same random candidates are used on every visit, so only your two sliders change the result.
A strict limit gives a smaller pool, but every member is far from the others. A loose limit gives a big pool full of cousins. Neither is free. You choose by what the model needs: range, or volume.
Self-Instruct tends to write easy, short requests, because the seeds are easy and short. Evol-Instruct, from the WizardLM paper, fixes this by rewriting an instruction step by step into a more complex one. People find it hard to write highly complex instructions at scale, so a model does the rewriting.
Here is one starting instruction, evolved three times. The moves are illustrative; the paper defines its own set.
| Step | Move | Instruction |
|---|---|---|
| 0 | start | Write a function that sorts a list. |
| 1 | add a constraint | Write a function that sorts a list of records by two keys, in place. |
| 2 | require reasoning | Write that function, and explain why your method stays stable when two records tie. |
| 3 | add an input case | Do the same for a list that does not fit in memory, and state the cost. |
Evolved instructions can also fail: the rewrite may become impossible, may lose the original point, or may only add filler words. So Evol-Instruct needs the same filter step, and it drops failed evolutions. In the WizardLM paper, human evaluators judged Evol-Instruct instructions better than human-created ones on a complexity-balanced test bed. The model finetuned on them reached more than 90% of ChatGPT’s capacity on 17 of 29 skills, by GPT-4 automatic evaluation.
Magpie takes a different route. As its title says, it prompts an aligned model with nothing. The idea in plain words: an aligned model is used to chatting, so given an empty prompt it can write the user’s request itself. Then you let it answer. No seed list is needed.
The paper reports prompting Llama-3-Instruct to generate 4 million instructions with their responses, then selecting 300K high-quality instances. A model finetuned from Llama-3-8B-Base on that data used SFT only, yet beat earlier public datasets that were used for both SFT and preference optimisation. For scale: the abstract says the official Llama-3-8B-Instruct was enhanced with 10 million data points through SFT, and that the Magpie model performed comparably to it.
The symptom is a pool of fifty thousand instructions where a human reader sees the same twenty ideas. The measurement is the nearest-neighbour similarity of each new record, logged in the record itself as nearest_sim. A healthy pool has a low average. A rising average says the generator is looping. The fix is to change what you show the generator: sample topics and skills from a list, not only earlier outputs, and tighten the limit.
For a pipeline that aims generation at a taxonomy of skills, see Simula: Reasoning-Driven Synthetic Data.
Chapter 2
Ask the model many times, keep only the answers a checker accepts, and train on what is left
A model solves a maths question correctly about three times in ten. That sounds weak. But you can ask the same question eight times. If any one of the eight answers is right, you now hold a correct worked solution that the model wrote itself.
This is the central trick of post-training data. A model is often better at recognising a right answer than at producing one on the first try. Checking is easier than solving. So you let the model solve many times, let a checker pick the winners, and teach the model from its own winning attempts.
Rejection sampling means: draw many candidates, reject those that fail a test, keep the rest. For training data the steps are these.
The data flow is simple. In: a list of prompts, each with a reference. Out: a list of records, zero or more per prompt. Here is one kept record. Notice that it stores how the verdict was reached, so a later audit can find out why it was kept.
json{
"prompt_id": "math-04417",
"sample_id": "math-04417/s5",
"response": "Let x be the price of a ticket. Then 3x + 6 = 42, so x = 12. The answer is 12.",
"verdict": {"checker": "exact_match", "extracted": "12", "reference": "12", "pass": true},
"k": 8,
"n_pass": 2,
"gen_model": "gen-v1"
}
Suppose the model solves a prompt with chance p on each independent try. The chance that all k tries fail is (1 − p) multiplied by itself k times. So the chance that at least one passes is called coverage, or pass@k:
p = 0.27: k = 1 gives 27.0% · k = 4 gives 1 − 0.734 = 71.6% · k = 8 gives 1 − 0.738 = 91.9%
Each step of the arithmetic: 0.73 squared is 0.5329. Squared again is 0.2840, which is 0.734. Squared once more is 0.0807, which is 0.738. So eight samples lift a one-in-four prompt to better than nine chances in ten. The cost is eight times the generation.
But p is not the same for every prompt. Try it with eight prompts that run from easy to very hard.
Each row is a prompt, easiest on top. The bar is the chance that at least one of k samples passes. The small number is the correct samples you expect to keep. Raise k, then try Show the answer.
Illustrative pass rates for eight prompts: 0.90, 0.70, 0.50, 0.35, 0.20, 0.10, 0.04, 0.01. The coverage formula is exact for independent tries. “Show the answer” is the STaR idea in a toy: a hint raises each pass rate by half of its gap to 1. The 600 tokens per sample is a made-up round number.
Watch the read-out as you raise k. Coverage of the easy prompts is full almost at once, and they flood the kept set. The hard prompts, which teach the most, need huge k. This is the main failure of the method, and the next sections show the fix.
STaR (“Bootstrapping Reasoning With Reasoning”) uses a small number of worked examples plus a large dataset with no worked reasoning. The model writes a rationale: the reasoning that leads to an answer. If the final answer is right, the rationale is kept. If it is wrong, the model tries again with the correct answer shown as a hint, and writes a rationale that reaches it. The model is then finetuned on the rationales that led to correct answers, and the whole loop repeats.
The retry with a hint is called rationalization. It rescues the hard prompts that plain sampling would never cover. In the paper, STaR significantly beat a model finetuned to predict final answers directly, and on CommonsenseQA performed comparably to finetuning a model 30 times larger.
There is a risk. A rationale written with the answer in view can reach the answer by a bad road: wrong steps that happen to land on the right number. The verifier checks only the end. So rationalized data needs the stronger checks of Chapter 5, or a judge that reads the steps.
The generate, verify, train loop is the backbone of post-training for large models. The Llama 3 report is the long public account of post-training at the 405B scale. We rely only on its abstract here: a dense Transformer with 405B parameters and a context of up to 128K tokens, with native support for coding, reasoning and tool use. The loop itself is general, and your own pipeline can follow it at any scale.
What the verifier is matters a great deal. A final-answer match is cheap and strict. Unit tests are strong for code. A model acting as judge is flexible and noisy; Chapter 8 shows how its holes become training signal. For how to measure a judge before you trust it, see Grader Reliability.
pythondef rejection_sample(prompts, gen, verify, k=8, max_keep=2):
kept = []
for p in prompts:
outs = gen(p["text"], n=k, temperature=1.0) # k raw candidates
passed = [o for o in outs if verify(p, o)["pass"]]
# cap per prompt so easy prompts do not flood the set
for o in passed[:max_keep]:
kept.append({"prompt_id": p["id"], "response": o,
"k": k, "n_pass": len(passed)})
return kept
The max_keep cap is the cheap fix for the flooding problem. Another is to weight hard prompts up: a prompt with n_pass of 1 out of 32 is far rarer than one with 30 out of 32. Log n_pass so you can do either later.
Chapter 3
Teach a model to call functions by running every generated call before it is allowed into the training set
A user types: “What is the weather in Oslo tomorrow?” An agent cannot answer from memory. It must call a tool, a function in the outside world. To do that, the model writes a small structured message and a program runs it.
That message is a function call. It names a function and fills in its arguments, usually as JSON. Training a model to write correct calls takes a large set of examples: many different functions, many ways to ask, and many ways to combine calls.
json{
"query": "What is the weather in Oslo tomorrow?",
"tools": [{"name": "get_forecast",
"parameters": {"city": "string", "days": "integer"}}],
"answer": [{"name": "get_forecast",
"arguments": {"city": "Oslo", "days": 1}}]
}
Writing such records by hand does not scale. A model can write them, but a model also makes quiet mistakes in them. The fix is to run the call. If the functions are real and executable, a wrong call shows itself.
APIGen is a pipeline for function-calling data that can be verified. The authors collected 3,673 executable APIs across 21 categories. A generator writes queries and calls for sampled APIs. Then every entry passes three hierarchical stages: format checking, actual function execution, and semantic verification. The released dataset has 60,000 high-quality entries.
Why three stages and not one? Each is cheaper than the next, and each catches a different kind of flaw. Run the cheap ones first so the costly ones see fewer records.
Here is a flaw that only the third stage can see. The user asks about Oslo. The generated call asks for Rome. The JSON is perfect. The function runs and returns a forecast. Every cheap test passes, and the record teaches the model to ignore the city.
Try the stages on a thousand generated candidates. The counts are illustrative. Switch the stages on and off, and watch which flaws remain in the kept set.
Each bar starts at the number of candidates of that kind. The solid part is what survives the checks you turned on. Start with all three off.
Illustrative batch of 1,000 candidates: 680 good, 70 with malformed JSON, 50 calling a function that does not exist, 110 that crash when run, 90 that run but answer a different question. Format and execution catch their flaws every time in this toy; with format off, execution still stops broken calls because they cannot run. The semantic judge catches 85% of the wrong-answer calls, because a model judge is imperfect.
With only the cheap checks on, the wrong-answer calls stay. Turning on the semantic stage removes most of them, but not all. The judge is a model, so it misses some. If you can, add a second route to the truth. For example, when the query names a city, check that the same city string appears in the call’s arguments.
Three design choices matter more than they look.
First, diversity of tools. A model trained on three hundred calls to one weather function learns that function, not tool use. The pipeline should sample tools from a wide pool, vary the argument types, and include queries that need more than one call, or no call at all. The 21 categories in APIGen are one way to force this spread.
Second, negative examples. Real users sometimes ask for things no tool can do. A model must learn to say so, not to invent a call. Include queries where the right answer is no call, and verify that the generator did not fabricate one.
Third, shape of the call. A real task often needs several calls: one after another, where the second uses the first result, or side by side, where they do not depend on each other. Include both, and verify each step. A record with three calls can fail at call two, and the stage field should say which one.
The reported result shows that this care pays. Models trained on the curated data reached state-of-the-art performance on the Berkeley Function-Calling Benchmark at 7B parameters, outperforming multiple GPT-4 models. The 1B model surpassed GPT-3.5-Turbo and Claude-3 Haiku.
pythondef verify_call(rec, tools, run, judge):
# stage 1: format (microseconds)
call = parse_json(rec["answer"])
if call is None or call["name"] not in tools:
return {"pass": False, "stage": "format"}
if not has_required_args(call, tools[call["name"]]):
return {"pass": False, "stage": "format"}
# stage 2: execution (needs a sandbox and a timeout)
ok, result = run(call, timeout_s=5)
if not ok:
return {"pass": False, "stage": "execution"}
# stage 3: semantics (a model judge, the costliest)
if not judge(rec["query"], call, result):
return {"pass": False, "stage": "semantics"}
return {"pass": True, "stage": "all", "result": result}
Look at what the function returns: not only pass or fail but the stage that failed. That small detail feeds the per-stage metrics of Chapter 7. When yield suddenly drops, the stage field says where to look.
For how agents turn exploration into training examples when the trace is long, not one call, see Trajectory Data.
Chapter 4
Break working code on purpose, and let the tests decide which broken versions make a good training task
A coding agent is trained on a simple loop: here is a repository with a bug, find it and fix it. Each training task needs a repository, a description of the problem, and a way to tell if the fix works. Where do thousands of such tasks come from?
The usual source is real GitHub issues that humans fixed. That supply is thin. The SWE-smith paper notes that earlier datasets were small, with at most thousands of training instances from 11 or fewer GitHub repositories. Making them took hundreds of hours of human work. The matching execution environments (a working copy of the repository with its dependencies installed) took several terabytes of storage.
SWE-smith turns the problem around. Instead of hunting for real bugs, start with a repository whose tests all pass, and make a bug. SWE-smith builds an execution environment for any Python codebase, then generates hundreds to thousands of task instances per codebase. The paper reports 50k instances from 128 GitHub repositories, which the authors call an order of magnitude more than previous work.
The recipe has five steps.
Step four is the filter. It is worth pausing on. The tests are the filter that separates a real task from a harmless edit. If the change breaks no test, either it did not change behaviour, or the tests cannot see it. Either way, the agent could never be graded on it, so the task is useless.
Here is a small function and six tests. Pick a way to break it and see what the tests say.
Choose a break. The changed line is marked. Each square is a test: green passes, red fails. A task needs at least one red square.
The function and the six tests are a toy written for this lesson, and the results are computed live by running the real arithmetic in your browser. The break types are a generic family for illustration; the SWE-smith paper defines its own methods.
Try “Use >=”. The code changed, but all six tests still pass. In this function, when the discount equals the cap, both versions give the same answer. This is an equivalent mutation: a change with no visible effect. It must be thrown away.
Try “Delete cap”. Only one test fails. That makes a valid but narrow task: one test to fix, five that must keep passing. Try “Flip >”: five tests fail, so the signal is loud.
Step five hides a trap. The model that writes the issue sees the failing test output, and the broken patch is nearby. If the issue names the exact line to change, the agent learns to read instructions, not to debug. Prompt the writer to describe the symptom as a user would see it, then spot-check a sample: if you can apply the fix from the text alone, the statement is too generous.
A task is not yet training data. The agent must attempt it. You run a strong agent on each task, many times, and keep the trajectories (the full record of its thoughts, commands and edits) that end with all tests passing. This is exactly the rejection sampling of Chapter 2, with the tests as the verifier. The paper trained SWE-agent-LM-32B this way, which reaches a 40.2% Pass@1 resolve rate on SWE-bench Verified, the best the authors report among open-source models.
The record keeps the grading split separate from the instruction, so any later stage can recompute the verdict.
json{
"instance_id": "shop__apply_discount.flip_cmp.3",
"repo": "shop", "base_commit": "a1b2c3d",
"break_patch": "- if off > cap:\n+ if off < cap:",
"fail_to_pass": ["t1", "t2", "t4", "t5", "t6"],
"pass_to_pass": ["t3"],
"problem_statement": "apply_discount charges the wrong amount for small discounts.",
"agent_pass_rate": "3/8"
}
Two lists define the grade. fail_to_pass holds the tests that were red and must become green. pass_to_pass holds tests that were green and must stay green, so the agent cannot fix the bug by breaking something else.
The first symptom is a model that scores high on your synthetic tasks and low on real ones. The measurement is a held-out set of real issues, scored side by side with a held-out set of synthetic ones. A big gap says the bugs you inject are not the bugs people make. The fix is to widen the break methods and to mix in some real issues.
The second symptom is a flat, easy dataset. Log agent_pass_rate for each task. Tasks that the agent solves 8 times of 8 teach little. Tasks it solves 0 of 8 may be broken or impossible. The middle band, such as 1 to 7 of 8, is where the learning is. Weight the mix towards it.
Chapter 5
Remove copies without erasing rare skills, and keep every test question out of the training set
Your pipeline has run for a night. It holds two million records that all passed their verifiers. A researcher opens a random page of them and sees the same phrase, “Write a function that…”, forty times in a row. Passing the check did not make the set varied.
Two more jobs remain after verification. First, dedup (short for deduplication): remove records that say the same thing. Second, decontamination: remove records that overlap with the evaluation questions you will use to measure the model. If you skip the second job, your scores lie.
Copies come in three strengths, and each needs its own tool.
Exact copies are identical text. Compute a hash, a short fingerprint of the text, and keep one record per fingerprint. This is fast and catches nothing subtle.
Near copies are the same text with small changes. A tool here is the n-gram: a run of n words in a row. Two texts that share many n-grams are near copies. Worked example with n = 4. Text A is “the cat sat on the mat today”. Text B is “the cat sat on the rug today”.
A has 4 four-grams. B has 4. They share 2 (“the cat sat on”, “cat sat on the”). Overlap = 2 / (4 + 4 − 2) = 2 / 6 = 0.33
One changed word breaks three of the four-grams that contain it. So n-gram overlap is strict: it catches copies that are almost word for word.
Meaning copies say the same thing in different words. For these, turn each record into an embedding: a list of numbers, say 768 of them, where texts with similar meaning get similar lists. Compare two lists with cosine similarity, the angle between them. Worked example in two dimensions: a = (3, 4) and b = (4, 3). The dot product is 3 × 4 + 4 × 3 = 24. Each vector has length 5. Cosine similarity = 24 / (5 × 5) = 0.96. A score near 1 means they point the same way.
The usual algorithm is greedy. Go through the records. Keep a record if no already-kept record is closer than your limit. Otherwise drop it. The limit is your main knob.
The danger is a limit that is too wide. Rare skills often come as a small tight group: eight records about the same unusual task. A wide limit sees them as copies of each other and keeps only one. You removed the thing you most wanted, a rare skill, and kept the common skills whole. Move the limit in the picture below and watch the small group at the bottom right.
Each dot is a record in embedding space; nearby dots mean similar. Faded dots were removed. The dashed ring marks a rare skill. Red rings are records that overlap the evaluation set.
Toy data, the same on every visit: six groups in a unit square with 90 records, 12 planted near copies, and 8 records marked as overlapping an evaluation set. Distance here is plain distance between dots, a stand-in for one minus cosine similarity.
At a limit of 0.01, the 12 planted near copies go (90 records become 77) and the rare group keeps 7 of its 8. Past 0.05 the rare group falls to 2. At 0.12 only 1 is left, and the big groups have thinned to a few dots each. The right limit sits just above the copy distance and well below the gap between real ideas. Find it by looking at pairs near the limit and reading them.
A benchmark is a fixed set of test questions. If a record in your training set contains a benchmark question or its answer, the model may memorise it, and the score no longer measures skill. Generated data is at risk, because the generator itself may have seen the benchmark on the web and may reproduce its questions.
The fix is to run the n-gram check of this chapter between every training record and every evaluation item. Pick a window length (a long one, such as 13 words, is an illustrative choice). If any window of that length in a record also appears in an evaluation item, drop the record. Switch on Remove eval overlaps above: the red rings disappear from the kept set, and a few good records go with them. That loss is the price of an honest score.
For the full treatment of filtering and dedup at web scale, see Data II: Filtering and Deduplication. For curation by embedding, see SemDeDup and DataComp.
Dedup answers “is it new?”. A quality score answers “is it good?”. Typical signals are the verifier’s pass rate, response length within a sane range, a judge model’s rating, and for code the number of tests the solution passes. A pipeline stores each score in the record, then keeps records above a limit. Keep the raw scores. The limit you choose today may be wrong next month.
pythondef dedup(records, embed, radius):
kept, kept_vecs = [], []
for r in records: # fixed order = repeatable
v = embed(r["text"]) # shape (768,), unit length
if all(1 - dot(v, w) >= radius for w in kept_vecs):
kept.append(r); kept_vecs.append(v)
return kept
def contaminated(text, eval_grams, n=13):
words = text.split()
return any(tuple(words[i:i+n]) in eval_grams
for i in range(len(words) - n + 1))
Chapter 6
Choose how much of each source to train on, then prove each source earns its place by taking it away
Your pipelines now produce four kinds of data: maths reasoning, code, tool calls and chat. You can train on a hundred thousand examples. Do you take twenty-five thousand of each? Or sixty thousand of code and ten of everything else? Nobody can tell you by thinking about it. You have to test.
The recipe for how much of each source goes into the training set is the mixture. Each source gets a weight: its share of the total. The weights add up to 1. Choosing them is a research problem of its own, because a good mixture can matter as much as a better generator.
Diminishing returns. The first ten thousand tool-call examples teach a lot. The next ten thousand teach less. A source that has already taught what it can adds almost nothing, and the same examples would do more good in a source that is still hungry.
Transfer. Sources help each other. Reasoning practice on maths may improve code. Code may improve tool use, because a tool call is a small piece of code. So the value of a source is not only its own score. It includes the lift it gives its neighbours.
Crowding out. The total is fixed. Every example you add to one source is one you take from another. Pushing the weight of one source up makes the others thinner, and a skill with too few examples slips.
An ablation is an experiment that removes one part to see what it was doing. A data ablation trains two models that differ in one source only, and compares their scores on the benchmarks. To keep the comparison fair, hold the total number of examples fixed: when you take out a source, give its share to the others.
The leave-one-out table is the standard view. For each source, train without it and record the change in every benchmark. Read the table in two ways. Down the diagonal: how much each skill depends on its own source. Off the diagonal: how much sources help each other.
Try it on a toy model. Set a mixture with the sliders, then use the selector to leave one source out. The bars show the benchmark scores. The thin line on each bar marks where that score was with all four sources in.
Set the weights. Bars are the benchmark scores that the mixture produces. The thin line is the score with all sources in.
Pure toy, not a result from any paper. Total fixed at 100,000 examples. Each score is a base plus a gain that saturates as the source grows (the first examples help most). Code gets 10% of the maths examples as help, and tool use gets 30% of the code examples as help. Weights you set are rescaled to add up to 100%.
Leave out code and you see two things. The code score falls hard: that is the diagonal. The tool score also falls a little: that is transfer. Now push the maths weight to its limit and the others to zero. The maths score rises, but each step buys less than the one before, while the other three fall away. That flattening is diminishing returns.
Every training run has luck in it: the order of examples, the random start. Train the same mixture three times and you get three slightly different scores. Say 61.0, 60.2 and 61.8. The mean is 61.0. Each run is 0.8 away on average (the spread, or standard deviation, is about 0.8). A new mixture that scores 61.5 is not better. It sits inside the noise.
So a data ablation needs repeats, or a gap much larger than the spread. Keep the benchmark set you tune on apart from the one you report. If you tune the mix on a benchmark and then quote the same benchmark, you have fitted the noise.
Full-size training for every mixture is too costly. Researchers train small models on small amounts of data to rank mixtures, then check the top few at full size. The ranking from small runs does not always carry over, so the final check is not optional. Record each mixture as a file, not as a memory:
pythonmixture = {
"id": "mix-0412",
"total": 100_000,
"weights": {"math": 0.30, "code": 0.30, "tool": 0.25, "chat": 0.15},
"sources": {"math": "ds-math-v7", "code": "ds-swe-v3",
"tool": "ds-call-v5", "chat": "ds-chat-v2"},
"seed": 17,
}
def build(mix, load):
out = []
for name, w in mix["weights"].items():
rows = load(mix["sources"][name]) # versioned dataset ids
take = round(mix["total"] * w)
out += random.Random(mix["seed"]).sample(rows, min(take, len(rows)))
return out
Notice the min(take, len(rows)). If a source is smaller than its quota, you silently get fewer examples than the weight says. A good mixer logs a warning when that happens. The weights you meant and the weights you got can differ.
Chapter 7
Turn one-off data scripts into composable stages with versions, provenance and numbers, so other researchers can build on them
Picture the lab three months later. Five researchers each run their own script. One of them reports the best model so far, and everyone asks: which data made it? The answer is a shrug. The generator changed twice, the filter threshold was edited in a notebook, and the output folder is called final_v2_really.
This is the common fate of data scripts. Each one works for its author. Together they are a mess that nobody can rebuild. A framework prevents this. It is a small set of rules, and a little code, that makes every pipeline the same shape. People adopt it when it saves them work on day one, and keep it when it saves them work on day ninety.
A stage takes a list of records and a config, and returns a new list of records and a dictionary of metrics. Nothing else. No hidden files, no global state. The six stages you met in this lesson fit the same shape.
| Stage | Does | Records in to out | Metrics it must log |
|---|---|---|---|
| Sampler | Picks task seeds, tools or repositories | nothing to seeds | seeds per skill, coverage |
| Generator | A model writes candidates | seeds to candidates | tokens spent, failures to parse |
| Verifier | Runs checks and attaches a verdict | candidates to verdicts | pass rate, rate per failure reason |
| Filter | Drops by score or rule | fewer records | drop rate per rule |
| Dedup | Removes copies and eval overlaps | fewer records | duplicates found, overlaps found |
| Mixer | Combines sources by weight | many sets to one | weights asked vs weights got |
Because every stage has the same shape, a new researcher writes a new verifier in twenty lines and drops it into the chain. Nobody else has to change anything. That is what composable means: parts that snap together.
pythonclass Stage:
name = "base"; version = "v1"
def run(self, records, cfg): # -> (records, metrics)
raise NotImplementedError
class Verify(Stage):
name = "verify"; version = "v2"
def run(self, records, cfg):
out, why = [], Counter()
for r in records:
v = check(r, timeout_s=cfg["timeout_s"])
if v["pass"]:
out.append({**r, "verdict": v, "trace": r["trace"] + [self.name]})
else:
why[v["stage"]] += 1
return out, {"n_in": len(records), "n_out": len(out), "dropped_by": dict(why)}
Look at what the stage does besides its job. It extends trace, the list of stages the record has passed. It records why records were dropped. These two habits cost almost nothing and are the reason the rest of the framework works.
Name each dataset by a content hash: a short fingerprint computed from everything that made it. That means the id of its input, the stage name, the stage version and the config. The same ingredients always give the same id. A changed ingredient gives a new id.
id(verify output) = hash( id(generator output), "verify", "v2", {"timeout_s": 5} ) = ds-7f3a
This does two jobs. First, the id is a version: if two researchers say ds-7f3a, they hold the same data. Second, the id is a cache key. Before running a stage, the framework checks if an output with that id already exists. If yes, it skips the work. Generation is the costly stage, so skipping it is the biggest win. Try it.
First press Run to build everything. Then pick a stage to change, which bumps its version, and press Run again. Purple boxes were reused. Warm boxes ran again.
Illustrative numbers: 10,000 seeds in; pass rates of 62% for the verifier, 80% for the filter, 85% for dedup and 90% for the mixer, each a little lower for every version bump. Cost units are made up as GPU-minutes: 1, 400, 60, 10, 30 and 5. The dataset id is a small hash computed live from the stage versions.
Change the mixer and only the mixer re-runs. The generator, the expensive part, is reused. Change the generator and everything after it re-runs, which is correct, because every later record came from it. That is the whole idea of a cache keyed by content: reuse exactly what is still valid.
Provenance means the record of where something came from. Each record keeps its parent ids, the generating model, the seed, the trace of stages and the verdicts. With these you can answer, for any surprising example in training, why it is there. You can also remove a whole family: if a generator version turns out to be bad, drop every record whose gen_model equals it.
A stage that only returns records hides its own behaviour. Require metrics: counts in and out, drop reasons, tokens spent, and one diversity number. Write them to a manifest next to the dataset. Then a sudden change in yield shows up as a line in a table, not as a mystery three weeks later. Compare two dataset ids and the framework should print what differs: which stage, which config, how many records.
yamlpipeline: swe-tasks-v3
stages:
- {name: sample, version: v1, cfg: {repos: repos-128, per_repo: 400}}
- {name: generate, version: v4, cfg: {model: gen-v1, temperature: 0.8, k: 8}}
- {name: verify, version: v2, cfg: {timeout_s: 5, require: [fail_to_pass]}}
- {name: filter, version: v1, cfg: {pass_rate: [0.125, 0.875]}}
- {name: dedup, version: v2, cfg: {radius: 0.01, decontam_ngram: 13}}
- {name: mix, version: v1, cfg: {weights: {swe: 0.7, tools: 0.3}}}
Notice that the whole recipe is a file, not code. Researchers share a file, review it like a change request, and run it. That is how a framework spreads through a team.
For the same ideas applied to a pipeline that generates data from reasoning about a taxonomy of skills, see Simula.
Chapter 8
Learn the three ways a healthy-looking pipeline goes bad, how to measure each one, and what to change
A pipeline can run for weeks, pass every check and fill a disk with clean-looking records, while it slowly makes the model worse. The failures are quiet. Each one has a symptom you can see, a number you can measure, and a fix. This chapter gives you all three for three failures.
Suppose a generator writes data, you train a new model on it, and that model writes the next round of data. Repeat. This is self-training. The paper “The Curse of Recursion” studies what happens. Its finding: use of model-generated content in training causes irreversible defects in the resulting models, and the tails of the content distribution disappear. The authors call this model collapse, and show it in variational autoencoders, Gaussian mixture models and LLMs.
The mechanism is easy to see with a bag of words. A tail is the set of rare things: unusual words, unusual tasks. Rare things have small probabilities. When you draw a finite sample, a rare thing may simply not appear. The next model never sees it, so it gives it probability zero. A thing with probability zero never comes back. Each round loses a few more, and the loss is permanent.
Worked example. A rare item has probability 0.5% (one in two hundred). You draw 200 samples. The chance that it never appears is 0.995 multiplied by itself 200 times, which is about 0.37. So in more than one round out of three, it vanishes. Over several rounds it is almost surely gone.
Forty kinds of item, commonest on the left. Each generation draws samples from the last generation’s distribution and fits the next one to them. The outline is the original. Drag the generations, then add fresh real data.
A real simulation of a toy, not a measurement of an LLM: 40 kinds of item with Zipf weights, categorical sampling with a fixed seed, and a new distribution fitted by counting. “Fresh real data” replaces that share of each round’s samples with draws from the original distribution. The paper reports no numbers for this; the toy only shows the mechanism.
Run it at 300 samples per round with no fresh data. By generation 12, many of the rarer kinds are gone for good, and the commonest ones hold more than their share. Now raise the sample size: loss slows, but does not stop. Now add fresh real data: the rare kinds that disappeared can return, because the real distribution still holds them. The paper makes the same point in words: data about genuine human interaction becomes more valuable as generated content spreads.
Measure it by counting distinct kinds of item or task in each generation and by tracking the nearest-neighbour similarity from Chapter 1. Fix it by keeping a real-data anchor in every round, by generating from several different models, and by never training only on the output of the model you are replacing. For the related loss of variety in one model’s answers, see Mode Collapse.
The verifier is the check that decides what trains the model. Anything the model can exploit in the check becomes a lesson. This is verifier leakage.
Examples: a maths checker that accepts any text containing the right number, so the model learns to list many numbers. A coding checker whose tests are visible to the generator, so it writes code that special-cases those tests. A judge model from the same family as the generator, so they share blind spots and agree with each other on wrong answers.
The symptom is a training score that climbs while a fresh, independent test does not. Measure it with canaries: put 100 known-bad records into the stream and count how many the verifier rejects. If it accepts 12 of the 100, its false accept rate is 12%, and every kept record carries that risk. Also hold out a second set of tests that the generator never sees. Fix it by closing the hole you find, using a different model family for the judge, and re-running the canaries after every verifier change. For judges, see Grader Reliability.
If benchmark questions leak into training data, the benchmark stops measuring skill. Generators can leak them without anyone intending it: a model that saw a benchmark on the web may reproduce its questions when asked for “a hard problem like this one”.
The symptom is a score that jumps with no matching gain on other tests. Measure it with the n-gram overlap check of Chapter 5, and with a paraphrase test: rewrite the benchmark questions with the same meaning and see if the score falls. A big fall says the model memorised the wording. Fix it by running decontamination as a fixed stage on every dataset version, with the eval set loaded fresh each time, and by logging the number of overlaps found.
| Failure | Symptom | Number to watch | Fix |
|---|---|---|---|
| Model collapse | Rare tasks vanish; answers sound alike | distinct kinds per generation | real-data anchor, many generators |
| Verifier leakage | Training score up, fresh test flat | canary false-accept rate | close the hole, hold-out tests, new judge family |
| Contamination | Benchmark jumps alone | n-gram overlaps; paraphrase drop | decontaminate every version |
pythondef canary_check(verify, bad_records, limit=0.02):
# records we KNOW are wrong; a healthy verifier rejects nearly all
accepted = sum(verify(r)["pass"] for r in bad_records)
rate = accepted / len(bad_records)
assert rate <= limit, f"verifier accepts {rate:.0%} of known-bad records"
return rate
Chapter 9
Keep the numbers, the terms, a checklist and the core code in one place, and find the sources to read next
A synthetic data pipeline is a chain of stages. A model writes candidates, checks remove the wrong ones, dedup removes the copies and the test questions, a mixer sets the proportions, and a framework gives every dataset an id, a history and a set of numbers. The generator is the cheap part. The checks, the filters and the framework decide the quality.
Every number below comes from the abstract of the paper named. Everything else in the lesson that carries a number is a toy and says so.
| Source | What it reports |
|---|---|
| Self-Instruct | 33% absolute improvement over vanilla GPT3 on Super-NaturalInstructions; a 5% absolute gap to InstructGPT-001 in human evaluation |
| WizardLM (Evol-Instruct) | More than 90% of ChatGPT’s capacity on 17 of 29 skills, by GPT-4 automatic evaluation |
| Magpie | 4 million instructions with responses generated, 300K selected; comparable to the official Llama-3-8B-Instruct, which was enhanced with 10 million SFT data points |
| STaR | Comparable to finetuning a 30 times larger model on CommonsenseQA |
| Llama 3 | A 405B dense Transformer, a context window of up to 128K tokens, native coding, reasoning and tool use |
| APIGen | 3,673 executable APIs in 21 categories; three checks (format, execution, semantics); 60,000 released entries |
| SWE-smith | 50k instances from 128 repositories; SWE-agent-LM-32B at 40.2% Pass@1 on SWE-bench Verified |
| The Curse of Recursion | Training on model-generated content causes irreversible defects; the tails of the distribution disappear |
| Term | In one line |
|---|---|
| Seed | A hand-written task that starts the pool |
| Rejection sampling | Draw k candidates, keep those that pass a test |
| Coverage (pass@k) | 1 − (1 − p)k: the chance that at least one of k samples passes |
| Verifier | A program that returns pass or fail for a candidate |
| fail_to_pass | Tests that were red and must turn green for a coding task to count as solved |
| n-gram | A run of n words in a row, used to find near copies and eval overlaps |
| Cosine similarity | The angle between two embedding vectors; near 1 means the same meaning |
| Mixture | The share of each source in the training set |
| Data ablation | Train with and without a source, compare scores, hold the total fixed |
| Content hash | A fingerprint of inputs, stage, version and config; it names a dataset and keys the cache |
| Provenance | The history stored inside a record: parents, model, seed, trace, verdicts |
| Model collapse | The permanent loss of rare cases from training on your own output |
pythonimport hashlib, json
def dataset_id(parent_id, stage, cfg):
blob = json.dumps([parent_id, stage.name, stage.version, cfg], sort_keys=True)
return "ds-" + hashlib.sha256(blob.encode()).hexdigest()[:4]
def run_pipeline(stages, cfgs, store):
parent, manifest = "seed", []
records = None
for stage, cfg in zip(stages, cfgs):
ds = dataset_id(parent, stage, cfg)
if ds in store: # cache hit: reuse the work
records, metrics = store[ds]["records"], store[ds]["metrics"]
metrics = {**metrics, "cached": True}
else: # cache miss: run the stage
records, metrics = stage.run(records, cfg)
store[ds] = {"records": records, "metrics": metrics}
manifest.append({"stage": stage.name, "id": ds, **metrics})
parent = ds
return records, manifest # the manifest IS the audit trail
Instruction synthesis (Chapter 1) and rejection sampling (Chapter 2) are the classic base. Verified tool-use data (Chapter 3) and test-graded coding tasks (Chapter 4) are how current agent and coding skills are built. Dedup, mixtures and the framework (Chapters 5 to 7) are what make the work repeatable. The failure modes (Chapter 8) are why every lab keeps real data and independent tests in the loop.
Further reading that this lesson does not rely on for any number: Tulu 3 (arXiv 2411.15124) and AgentInstruct (arXiv 2407.03502).
Now press Present or Teach and explain one pipeline from memory: where the seeds come from, what each check catches, why the mixer needs ablations, and how the content hash saves a rerun. Then go back to the filter at the top of the page and drag the verifier’s strength again.