A new data mix scores 1.1 points higher on one run. Learn to tell a real gain from noise, with seeds, paired tests, power and honest baselines, before you change a post-training recipe.
An ablation is a controlled experiment: change one thing in a training recipe, hold everything else fixed, and measure what moved. Post-training is the stage after pretraining that turns a base model into an assistant.
Press Run the experiment a few times with one run per arm, then switch to ten runs. Count how often the better mix looks worse. Then we build, piece by piece, the habits that stop a lucky run from steering a team.
You need a rough idea of an average and a little Python for the code. We build the statistics from zero.
Built from eight open papers, from Deep Reinforcement Learning that Matters to Signal and Noise. All sources are listed in the last chapter.
Illustrative: the baseline truly scores 60.0% and the new mix 60.5%, a real gain of 0.5 points. Each run adds luck with a spread of 1.5 points. These numbers are made up to show the effect; Chapter 1 gives measured magnitudes from the literature.
Chapter 0
See how a single training run can make a useless change look like a win, and why that matters before a team builds on it
It is Monday. Your team has two versions of a data mix for post-training. Version 6 is the current one. Version 7 adds more code and math examples. You train a model on each and score both on the same set of questions. Version 7 gets 61.4%. Version 6 gets 60.3%.
That is a gain of 1.1 points. Someone says "ship it". Someone else says the next three experiments should build on version 7. Nobody asks the one question that matters: if we ran this again, would version 7 still win?
This lesson is about that question. We will build the habits that answer it, one at a time, with small toys you can play with and the Python you would write.
Post-training is the work done after pretraining. Pretraining teaches a model to continue text. Post-training turns it into an assistant, with example conversations, preference data and reward signals. Teams change this recipe all the time: a new data source, a new filter, a different learning rate.
An ablation is an experiment that tests one such change. You build a model with the change and a model without it. You score both. The gap is your evidence. The model without the change is the baseline, also called the control. Each trained model is one run.
To score a run, you give it an eval: a fixed list of questions, called items, each marked right or wrong. The score is the share of items it gets right. Think of a class test. The score is real, but it depends on which questions were on the test and on how the student slept.
Training a model has luck in it. The first weights are random. The examples arrive in a shuffled order. A different seed, which is the number that starts the random generator, gives a different run. The recipe is the same, yet the score moves.
So your 1.1 points holds two things mixed together: the true effect of version 7, and the luck of two particular runs. One number cannot split them. A researcher on a frontier post-training team learns early to ask how big the luck is before reading the gap.
The classic warning comes from reinforcement learning. Henderson and colleagues (arXiv 1709.06560) found that results in deep reinforcement learning are hard to reproduce. Nondeterminism in the test environments, plus variance inherent to the methods, made reported gains hard to interpret. They asked for significance metrics and standard reporting. The same warning holds for language models.
Suppose the luck in one run is a spread of 1.5 points. That number is illustrative here; Chapter 1 shows measured spreads. The gap between two single runs then has a spread of about 2.1 points, because both runs bring their own luck.
Now imagine a team that tests twelve ideas, and none of them does anything. Each idea still produces a gap. Some gaps are positive by chance. If the team ships anything above +1 point, a surprising share of the useless ideas passes. Try it below.
Each bar is the measured gap of one idea that truly changes nothing. Red bars pass your shipping bar. Drag the bar up or down, then draw twelve new ideas.
Illustrative: every idea has a true effect of exactly 0. Each run carries luck with a spread of 1.5 points, so the gap of two single runs has a spread of 2.1 points. The share that passes is computed from that normal curve.
At a bar of +1 point, about one useless idea in three ships. The team then builds the next experiment on top of it. Two more such steps and the recipe holds three changes that do nothing, plus whatever harm they hide. Raising the bar helps. But a higher bar also throws away real gains, and we want a better answer than a guess.
The better answer is to measure the luck, so we know the size of a gap we can trust. That is what the rest of the lesson does. The hero at the top showed the fix in one move: more runs per arm shrink the luck. Chapter 3 makes that exact.
Chapter 1
Name the five sources of run-to-run spread, and see how they add up to the error bar on every score
Imagine two cooks who follow one recipe for bread. Both loaves differ a little. The oven runs warm in one corner. One cook kneads a bit longer. The flour came from two different bags. None of this is a different recipe, yet the loaves are not twins.
Training a model is the same. This chapter lists the places where the "same recipe" quietly differs. Once you can name them, you can measure them, and a measured spread is an error bar.
Initialization. Some weights start as random numbers. A new seed gives new starting weights, so training walks a different path. In post-training, the base model is usually fixed. Still, new layers, such as a reward head or a LoRA adapter, start at random.
Data order. Training sees examples in batches. The shuffle decides which examples meet in the same batch and which come early. Early and late examples shape the model differently. A new order is a new run.
Sampling during training. Dropout masks, and for reinforcement learning the model's own samples, use random numbers too. Each of them adds spread.
Nondeterministic kernels. A kernel is a small program that runs on the GPU. Many kernels add numbers in an order that changes from launch to launch. Computer arithmetic is not exactly associative, so a different order gives a slightly different sum. Two runs with identical seeds can still drift apart. Our lesson on defeating nondeterminism shows how to remove this one.
Eval sampling. This one is different, because it does not come from training at all. Your eval is a finite list of items. A different list of the same kind would give a different score, even for the same model. We call this the luck of the draw of the questions. Chapter 3 gives its exact size.
Dodge and colleagues (arXiv 2002.06305) fine-tuned BERT on four GLUE datasets. They ran hundreds of trials per dataset and changed only the random seed. They released the scores of 2,100 trials. Their abstract says that identical settings with distinct seeds can give substantially different results. It also says that weight initialization and data order contribute comparably to the variance.
Madaan and colleagues (arXiv 2406.10229) asked the same question for language model benchmarks. They measured seed variance across initializations, and also how consistent scores are during training. They found that turning a multiple-choice task into a completion task can often lower variance for models near 7B parameters. Standard tools from human testing, such as item response theory, did not reduce it much.
Both papers give one message. The spread is not a footnote. It is large enough to decide which of two recipes looks better. The exact size depends on your task, model and eval, so you must measure it on your own setup. We will show how.
Independent sources of luck add in a special way. You do not add their sizes. You add their variances. The variance is the square of the spread, where the spread (the standard deviation) is the typical distance of a score from its average. In symbols, for independent sources:
√(0.82 + 0.82 + 1.12 + 0.32) = √(0.64 + 0.64 + 1.20 + 0.09) = √2.57 = 1.6 points (illustrative numbers)
Two things follow. The largest source dominates, because squares grow fast. And removing a small source does little. Try it: switch the sources on and off, and watch 400 imaginary runs of one recipe pile up.
Each bar counts how many of 400 runs of one recipe landed in that score bin. Press a source to add its luck. Watch the pile widen.
Illustrative spreads in points, not measurements. The eval term is the exact size for 2,000 items at a 60% score (Chapter 3). The 400 runs use fixed random numbers, so switching a source only scales its part.
Notice what the eval term does. Even if you could train with zero luck, you would still see about 1.1 points of spread from the item draw. Only a larger eval, or a paired comparison, removes it. Those are Chapters 3 and 4.
You cannot study a source you did not record. Give each source its own seed and write all of them in the run record. Then a seed sweep can vary one at a time.
pythonimport json, random, numpy as np, torch def set_seeds(init_seed, data_seed): random.seed(init_seed); np.random.seed(init_seed) torch.manual_seed(init_seed) # initialization, dropout g = torch.Generator().manual_seed(data_seed) # data order only return g # pass to DataLoader(generator=g) record = {"run_id": "mix-v7-s2", "recipe": "mix-v7", "init_seed": 2, "data_seed": 2, "eval": "evalset-v3", "n_items": 2000, "deterministic_kernels": False, "score": 0.612} # to isolate data order: hold init_seed fixed, sweep data_seed (and the reverse) print(json.dumps(record))
Chapter 2
Build an experiment where the only difference between two runs is the thing you are testing
A friend says her new running shoes made her faster. You ask what else changed. She also trained twice as often, slept better, and ran on a flat track instead of a hill. Now you cannot say what the shoes did. The shoes were mixed up with other changes.
Researchers call this mixing a confound: a second difference that travels with the first and takes credit for its effect. A controlled ablation exists to remove confounds. It has three rules, and each one is simple to state and easy to break.
Take the baseline recipe. Change exactly one field. Keep everything else byte for byte the same: the model, the learning rate, the number of steps, the seeds, the eval. If you want to test two changes, run three more arms: A alone, B alone, and A with B. Then you also learn whether they help each other.
Teams break this rule by accident. A new data mix comes with a new filter, a new template and a new deduplication pass. Each is a change. If you cannot separate them, say so in the write-up, and test them one by one when the result matters.
You can make the rule a test in code. Write each arm as a config, and check that the two configs differ in one field before any GPU time is spent.
pythondef diff(a, b): return {k: (a.get(k), b.get(k)) for k in a.keys() | b.keys() if a.get(k) != b.get(k)} base = {"data_mix": "v6", "lr": 1e-5, "steps": 3000, "batch": 128, "eval": "evalset-v3"} new = {"data_mix": "v7", "lr": 1e-5, "steps": 3300, "batch": 128, "eval": "evalset-v3"} changed = diff(base, new) assert list(changed) == ["data_mix"], changed # fails: {'data_mix': ('v6','v7'), 'steps': (3000, 3300)}
The assertion fails here, and that is the point. Someone gave the new run 300 more steps. Without the check, nobody would have noticed.
The baseline is not a straw man. It must be the best version of the old recipe that you can build with the same effort. If the new recipe got a sweep over four learning rates and the baseline got one, then part of the gap is the sweep, not the idea.
The Deep Reinforcement Learning that Matters paper (arXiv 1709.06560) illustrates how much reported metrics and results vary when you compare against common baselines. That is the lesson here: the baseline's setup is part of the result. Give both arms the same tuning budget, and write the budget down.
Also re-run the baseline in your own pipeline. A number copied from an old table was made with an older eval script, a different seed set, maybe a different tokenizer. It is not the same measurement.
Compute is the work a run spends: how many tokens it saw, for how many steps, at what cost in FLOPs (the count of arithmetic operations). More compute usually means a better model, for any recipe. So a run that spends more can win without a better idea.
The fix is to match compute between arms. That means the same number of training tokens and steps. If the new data has longer examples, the same number of examples costs more FLOPs, so you must choose which to match and say which. Often the honest choice is to match FLOPs, and to show the token count next to it.
Here is a worked example with made-up scaling. Suppose each doubling of training tokens adds 2.0 points, and the new mix truly adds 0.4. The new run got 30% more tokens, and the team tuned its learning rate but not the baseline's, worth 0.5 points. The gap is the sum of the parts:
gap = 0.4 (mix) + 2.0 × log2(1.3) (tokens) + 0.5 (tuning) = 0.4 + 0.76 + 0.5 = 1.7 points (illustrative)
Only 0.4 of the 1.7 belongs to the idea. Take the pieces apart yourself.
The teal part is the data mix. The yellow parts are compute and tuning that leaked in. Give the new run more tokens, then switch on matching and equal tuning.
Illustrative model, not a measurement: 2.0 points per doubling of tokens, 0.5 points from extra tuning, 0.4 points from the mix. Real scaling is not this tidy, which is why you match compute rather than correct for it.
When both switches are on, the gap shrinks to the true 0.4. You can never read that number off a single comparison with unmatched arms. You can only get it by design.
Open post-training reports such as Tulu 3 (arXiv 2411.15124) and OLMo 2 (arXiv 2501.00656) release their data, code and recipes. That lets others rerun the comparison with matched settings. Treat that as the standard: a result you cannot rerun is a claim, not a measurement.
Chapter 3
Turn the luck into a number, the standard error, and use it to plan an experiment that can actually see the effect you care about
A news poll asks 1,000 people whether they like a new law. Sixty percent say yes. The report adds "plus or minus 3". Nobody thinks the true share is exactly 60. The poll saw a small sample, and a different sample would give a slightly different number.
An eval is a poll of questions. Your model answers 1,000 items and gets 60% right. A different set of 1,000 items of the same kind would give another score. The size of that wobble has a formula, and the formula tells you how big your eval must be.
Each item is right or wrong. Call the chance of "right" p. The score over n items has a spread, called the standard error (SE): the typical distance between your score and the true score you would see on endless items.
p = 0.6, n = 1,000: SE = √(0.6 × 0.4 / 1,000) = √0.00024 = 0.0155, which is 1.55 points
Miller (arXiv 2411.00640) argues that eval questions are a draw from an "unseen super-population" of possible questions, and gives formulas for analyzing evals, comparing two models and planning eval experiments. The SE above is the simplest of those formulas. Real evals can have clusters of related items, which make the true SE larger.
You compare two models, A and B. Each has its own SE. The gap has a bigger one, because both scores wobble. If A and B are independent, their variances add:
SEgap = √(1.552 + 1.552) = 1.55 × 1.414 = 2.19 points (two 1,000-item scores near 60%)
A 95% confidence interval is the gap plus or minus 1.96 SE. Here that is plus or minus 4.3 points. A measured gap of 2 points on this eval is inside it. The data cannot tell 2 points from zero.
Training luck (Chapter 1) averages out when you train several seeds. The spread of the mean of k runs is the spread of one run divided by the square root of k. Four seeds halve it. But the eval term does not shrink with more seeds, because every seed is scored on the same items. Only more items shrink it.
SEarm = √( p(1 − p)/n + σtrain2/k ), and SEgap = √2 × SEarm
Before you spend GPU time, ask: if the new mix truly helps by d points, will this experiment notice? The chance that it does is its power. A common target is 80% power with a 5% false alarm rate. For those targets, the minimum detectable effect (MDE) is the true gain you would catch four times out of five:
MDE = (1.96 + 0.84) × SEgap = 2.8 × 2.19 = 6.1 points (1,000 items, one run per arm, no training luck)
Six points! An experiment like that can only see large effects. Most recipe changes add one or two points. To see a 2 point effect with 80% power, solve for n with p = 0.6:
n = 2.82 × 2 × 0.24 / 0.022 = 7.84 × 0.48 / 0.0004 = 9,408 items per arm
That is nearly ten thousand items, and it ignores training luck. The picture below shows the idea. The grey curve is the gap you would measure if nothing changed. The teal curve is the gap you would measure if the true effect were d. The green area is the share of the teal curve past the bar for a false alarm. That share is your power.
Choose the eval size, the seeds per arm and the true gain you hope to see. The green area is your power. Make it reach 80%.
Normal approximation, two-sided 5% false alarm rate, a 60% baseline score, and training luck of 1.2 points per run (illustrative). Items are treated as independent and the two arms as unpaired. Chapter 4 shows how pairing shrinks the spread.
Two sliders matter most. Items fight the eval term; seeds fight the training term. When one term dominates, adding the other does little. Check which term is larger before you buy more of either.
pythonimport numpy as np from scipy.stats import norm def se_gap(p, n, sigma_train, k): se_arm = np.sqrt(p * (1 - p) / n + sigma_train**2 / k) # fractions, not points return np.sqrt(2) * se_arm def mde(se, alpha=0.05, power=0.8): return (norm.ppf(1 - alpha / 2) + norm.ppf(power)) * se print(mde(se_gap(0.6, 1000, 0.0, 1))) # 0.0613 -> 6.1 points print(mde(se_gap(0.6, 9408, 0.0, 1))) # 0.0200 -> 2.0 points
Chapter 4
Score both models on the same questions and look only at where they disagree, which can cut the error bar by more than half
Two students take the same exam. Student A gets 600 of 1,000 questions right. Student B gets 620. Is B better? Most of the questions were easy, and both got them right. Most of the rest were very hard, and both got them wrong. Those questions say nothing about who is better.
The questions that carry the signal are the ones where the students differ. If B wins nearly all of those, B is better. If the split is close to even, the 20 point lead is luck.
Chapter 3 treated the two scores as independent polls. That ignores a fact: both models answered the same items. Hard items are hard for both. The item draw, which caused much of the error bar, hits both models in the same way, so it largely cancels in the difference.
A paired comparison uses this. For each item, compute the difference between the two models' results. Then treat the list of differences as your data. The item-draw luck that both models share drops out.
Take 1,000 items. Model A is right on 600, model B on 620. Sort each item into one of four boxes:
| Outcome | Items | Difference B minus A |
|---|---|---|
| Both right | 570 | 0 |
| Only A right | 30 | −1 |
| Only B right | 50 | +1 |
| Both wrong | 350 | 0 |
Check: A is right on 570 + 30 = 600 items, and B on 570 + 50 = 620. The mean difference is (50 − 30) / 1,000 = 0.020, which is 2.0 points. Now the spread of the differences. Each difference is −1, 0 or +1, so the average of the squares is the share of items that disagree, (30 + 50) / 1,000 = 0.080. The variance is that, minus the squared mean:
variance = 0.080 − 0.0202 = 0.0796, so SE = √(0.0796 / 1,000) = 0.00892, which is 0.89 points
The unpaired answer from Chapter 3, for the same scores, is √(0.24/1,000 + 0.2356/1,000) = 0.0218, which is 2.18 points. The paired SE is 2.4 times smaller. In z-scores, the paired gap is 2.0 / 0.89 = 2.24, and the unpaired gap is 2.0 / 2.18 = 0.92. The same data is a clear result with pairing and a shrug without it.
For right-or-wrong items there is a classic test on exactly these counts. McNemar's test looks only at the 30 and the 50. Its statistic is (50 − 30)2 / (50 + 30) = 5.0, a p-value near 0.025. The z-score above is its square root.
Now play with it. Keep the two scores fixed at 60% and 62%. Slide the number of disagreeing items. Two models that make similar mistakes disagree on few items, so they pair well.
The top bar sorts 1,000 items into four boxes. Below it, two error bars for the same 2.0 point gap. Drag to change how many items only A gets right.
Toy counts, with A fixed at 600 right and B at 620 right out of 1,000. B-only items are always 20 more than A-only items. Intervals are the gap plus or minus 1.96 standard errors, from the formulas in the text.
At the far right, the models disagree on almost everything, and pairing is worse than treating them as independent. That is rare in practice. Two models fine-tuned from the same base usually share most of their answers, so pairing usually wins big.
The same idea works when scores are graded instead of right or wrong. Take the per-item score difference, then its mean and standard error. Miller's paper (arXiv 2411.00640) gives formulas for comparing two models and for planning evals in this spirit. Our lessons on eval statistics and error bars on evals go further.
This works only if you keep per-item results. Do not save just the final score. Save one record per item, keyed by an item id that is the same for every model.
python# results_A.jsonl / results_B.jsonl: {"item_id": "q0173", "correct": 1} import json, numpy as np def load(path): return {r["item_id"]: r["correct"] for r in map(json.loads, open(path))} A, B = load("results_A.jsonl"), load("results_B.jsonl") ids = sorted(A.keys() & B.keys()) # same items, same order d = np.array([B[i] - A[i] for i in ids]) # shape (n,), values -1, 0, +1 gap, se = d.mean(), d.std(ddof=1) / np.sqrt(len(d)) print(f"gap {gap:+.3f} 95% CI [{gap-1.96*se:+.3f}, {gap+1.96*se:+.3f}]")
Chapter 5
Test whether the winner of a cheap small-scale run is also the winner at full scale, and count how often the ranking flips
A bakery wants to pick the best of 25 cake recipes for a wedding of 1,000 guests. Baking 25 full cakes costs too much. So the baker bakes 25 cupcakes, tastes each, and picks the winner. This works only if a cupcake that tastes best also makes the best big cake.
Frontier teams face this choice all the time. A full-size run costs weeks of a large cluster. A small run costs minutes. If the small run ranks recipes the same way as the big run, you can explore cheaply and spend big compute once. If it does not, you pick the wrong recipe, with confidence.
Magnusson and colleagues built a test bed for exactly this question (arXiv 2504.11393). They trained models on 25 different pretraining corpora, which differ in their sources, deduplication and filtering. The models range up to 1B parameters, with up to 100B tokens, and 3 random seeds. They released the models, data and evals.
Their headline finding is simple. Rank the 25 corpora by the score of a model with 150M parameters. That ranking predicts the better corpus at the 1B target scale for about 80% of pairs of corpora. They also tried 8 scaling-law methods, which fit curves across several small sizes. None of those 8 baseline methods beat the compute-decision frontier of single-scale predictions.
One more result matters for evals. Using continuous likelihood metrics at small scale, instead of the usual accuracy, made MMLU, ARC, HellaSwag, MBPP and HumanEval more than 80% predictable at the 1B scale, using just 0.01% of the target compute. We return to why metric choice matters in Chapter 6.
Note the scope. DataDecide studies pretraining data. Post-training recipes, such as the mixes in Tulu 3 (arXiv 2411.15124), may transfer differently. Treat the study as the best open evidence for the method, and check it for your own setting.
The measure is pairwise accuracy. Take every pair of recipes. For each pair, ask: does the small run pick the same winner as the big run? The share of pairs where it does is the pairwise accuracy. With 25 recipes there are 25 × 24 / 2 = 300 pairs, so 80% means 240 correct pairs.
A tiny worked case with three recipes:
| Recipe | Small-scale score | Large-scale score |
|---|---|---|
| A | 41 | 55 |
| B | 40 | 58 |
| C | 38 | 52 |
The pairs: A against B, small picks A, large picks B, so wrong. A against C, both pick A, right. B against C, both pick B, right. Two of three pairs are right, so the pairwise accuracy is 67%. Notice the wrong pair is the one with the closest small-scale scores, 41 and 40. Close calls flip first.
Small runs have their own luck. Each small score is the true quality plus noise. When the noise is larger than the gap between two recipes, their order is a coin flip. Below, 25 imaginary recipes have a true large-scale quality. The left column is their small-scale order and the right column is their large-scale order. A red line is a recipe that moved five places or more.
Each line joins one recipe's rank at small scale to its rank at large scale. Crossing lines are pairs the small run got wrong. Raise the noise, or switch to a quieter metric.
Toy model, not DataDecide's data: 25 recipes whose true large-scale scores are spread over about 12 points, plus Gaussian noise at small scale. DataDecide measured about 80% at 150M parameters for 1B, from its 25 corpora. The quieter metric halves the noise for illustration only.
The accuracy depends on the noise, but also on the gaps. Recipes that differ by a lot are almost never swapped. Recipes that differ by a hair often are. That is fine if you only need to find a clearly better recipe. It is a problem if you want to rank a crowd of near-equals.
A practical rule follows. Use small runs to throw away clear losers and to shortlist a few. Then confirm the finalists at the scale you care about, with the seeds and paired checks from Chapters 3 and 4. And before you trust the method, run two or three recipes at both scales and see whether their order agrees.
pythonfrom itertools import combinations def pairwise_acc(small, large): # dicts: recipe -> score pairs = list(combinations(small, 2)) right = sum((small[a] - small[b]) * (large[a] - large[b]) > 0 for a, b in pairs) return right / len(pairs) small = {"A": 41, "B": 40, "C": 38} large = {"A": 55, "B": 58, "C": 52} print(pairwise_acc(small, large)) # 0.667
Chapter 6
Measure how well an eval separates good models from bad, and how much it wobbles on its own, before you let it make a decision
You step on a bathroom scale. It reads 70.2 kg. You step on again. It reads 73.1 kg. Then 68.9 kg. The scale wobbles by about 2 kg every time. Could you notice that you gained half a kilo this week? No. The scale is too noisy for that question.
A benchmark is a scale for models. Some benchmarks are steady. Others jump around even when the model barely changes. Before you use one to settle an ablation, you should know which kind it is.
Heineman and colleagues (arXiv 2508.13144) give names to the two numbers. Signal is how well a benchmark separates better models from worse ones. Noise is how sensitive the score is to random variability between training steps. They studied 30 benchmarks and 375 open-weight models, from 60M to 32B parameters, and released about 900K evaluation results.
Here is a plain version you can compute. Take a set of models that really differ. The signal is the spread of their scores. Then take one model and read its score at the last several checkpoints, which are saved copies of the model from nearby training steps. The noise is the spread of those scores. The ratio is the signal-to-noise ratio, SNR.
This is a simplified form of the paper's idea, written so you can compute it in ten lines. Read the paper for its exact definitions. The key claim holds either way: the paper argues for benchmarks that are more reliable for small-scale decisions, and it tests ways to raise signal or cut noise.
Take six models scoring 44, 46, 48, 51, 53 and 56 on a benchmark. The mean is 49.7. The squared distances from it are 32.1, 13.5, 2.8, 1.8, 11.1 and 40.1. Their sum is 101.4. Divide by 5 and take the root: the signal is 4.5 points.
Benchmark X wobbles by 0.9 points between checkpoints. Its SNR is 4.5 / 0.9 = 5. Benchmark Y wobbles by 3.0 points, so its SNR is 1.5. Now ask how often each benchmark puts two models 2 points apart in the right order. The score of each model has noise σ, so the difference has noise √2 × σ. The chance of the right order is the normal curve at 2 / (1.414 × σ):
X: Φ(2 / (1.414 × 0.9)) = Φ(1.57) = 94%. Y: Φ(2 / (1.414 × 3.0)) = Φ(0.47) = 68%
On the steady benchmark you get the order right about 19 times in 20. On the noisy one, about two times in three. Same models, same gap, a different tool.
The paper tests three interventions. Each one is cheap.
Madaan and colleagues found a related effect (arXiv 2406.10229): turning a multiple-choice task into a completion task often reduces variance for models around 7B. Both papers say the same thing in different words. Choose the measurement that is quiet before you choose the model that is loud.
Each column is one model. Each dot is one reading of its score. When columns overlap, the benchmark cannot order those models. Switch benchmark, average checkpoints, or smooth the metric.
Toy numbers: six models at 44, 46, 48, 51, 53 and 56, so the signal is 4.5 points. Benchmark noise is 0.9 or 3.0 points per reading. Averaging k checkpoints assumes independent noise, which real checkpoints only partly satisfy. The 0.6 factor is an illustration, not a measured value.
pythonimport numpy as np # final[m]: score of model m. last[m]: its scores at the last 10 checkpoints. signal = np.std(list(final.values()), ddof=1) # spread across models noise = np.mean([np.std(last[m], ddof=1) for m in last]) # wobble within a model snr = signal / noise smooth = {m: np.mean(last[m][-5:]) for m in last} # average last 5 checkpoints
Chapter 7
See how choosing the best run, the best checkpoint or the best metric after the fact builds a gain out of nothing
A student takes a test ten times and hands in the best score. Another student takes it once. Whose score tells you more about what they know? The first student's score is a maximum, and a maximum of ten noisy tries is almost always higher than a typical try. It measures luck as much as skill.
Ablations fall into this trap quietly. The trap has three forms: pick the best seed, pick the best checkpoint, and test many ideas and report the winner. All three give a gain that disappears when someone reruns the experiment.
A training curve plots a score against training steps. A single curve per arm hides the luck. Plot every seed as its own thin line, and the mean as a thick one. Then ask three plain questions.
Stopping early is fine when a rule you set in advance does it, using a dev set (data held out for decisions) and not the final test. It is a cherry-pick when you stop at the step where your favorite arm looks best.
Dodge and colleagues also measured how the best-found performance changes with the number of trials. More trials give a higher maximum, even though the recipe did not change. Take a recipe with run-to-run spread σ. The expected best of k runs sits above the true score by a multiple of σ:
| Runs, k | Expected best, in spreads above the truth | With σ = 1.5 points |
|---|---|---|
| 1 | 0.00 | +0.0 |
| 5 | 1.16 | +1.7 |
| 10 | 1.54 | +2.3 |
| 100 | 2.51 | +3.8 |
These are standard facts about the maximum of normal draws. They are not from the papers. They show the size of the effect: with ten tries and a spread of 1.5 points, a recipe that does nothing looks 2.3 points better. Try it.
Every run here is the same recipe with a true score of 60. Grey shows single runs. Yellow shows the best of k runs. Raise k.
Illustrative: run-to-run spread of 1.5 points, 600 imaginary experiments with fixed random numbers. Checkpoints of one run are correlated, so picking the best checkpoint inflates less than picking the best of independent seeds, but the direction is the same.
The cure is a rule: decide the number of seeds and the stopping step before you run. Report the mean and the spread across all seeds, never the best one. If you ran twelve and show eight, say why.
Now widen the view. Suppose you test 20 ideas that all do nothing. Each test has a 5% false alarm rate. The chance that none raises a false alarm is 0.95 to the power 20, which is 0.36. So with probability 64% you will see at least one "significant" idea. This is the multiple comparisons problem.
P(at least one false alarm) = 1 − 0.95m; for m = 20: 1 − 0.358 = 64%
The problem is bigger than it looks, because the 20 are often hidden. You looked at three metrics, four eval subsets, two checkpoints and three seed sets. That is 72 ways to cut the same data, and you only reported the cut that looked good. Statisticians call this the garden of forking paths. Each fork was a reasonable choice. Together they are a search.
Two fixes work together. First, write the analysis plan before the run: one primary metric, one checkpoint rule, one seed set, one decision threshold. Everything else is labeled exploratory. Second, when you must test many things, tighten the bar. The simplest tightening is Bonferroni's rule: use 0.05 / m as the bar for each test. It is blunt, and it keeps the total false alarm rate near 5%.
Each dot is the p-value of one idea that truly does nothing. A p-value is the chance of seeing a gap this big by luck alone. Red dots fall under the bar and look like wins. Press the button for a new batch.
Every idea has a true effect of exactly zero, so each p-value is a uniform random number between 0 and 1. The bar is 0.05, or 0.05 divided by m with Bonferroni. The chance of at least one false alarm is computed from these rules.
Chapter 8
Spot confounded changes, leaked eval items and a test set that was tuned on, each by its symptom, its measurement and its fix
A teacher gives a class a practice sheet. Later, the real exam has the same questions. The class scores very high. Did they learn the subject, or did they memorize the sheet? From the score alone, you cannot tell.
Ablations can fail in three ways that look like success. For each, the symptom is a gain that seems too good, or that vanishes on a rerun. Then you need a measurement that separates the real from the fake, and a fix.
Chapter 2 covered this one. The symptom is a gain that comes with a second change: more tokens, more tuning, a new filter, a new chat template. The measurement is a config diff that lists every field that differs, plus a re-run with matched compute. The fix is to change one thing, or to add arms that isolate each change.
Contamination means that eval items, or close copies of them, appear in the training data. The model then remembers answers, and the score measures memory, not skill. Post-training mixes are built from many open datasets, and some of those contain benchmark questions.
The symptom is a gain on one benchmark with no gain on a different one that tests the same skill. The measurement is to count the overlap. Search the training data for each eval item: exact string matches first, then near matches, such as the same question with different spacing or a paraphrase. The fix is decontamination: remove the matching training examples, or drop the matching eval items from the comparison.
Tulu 3 (arXiv 2411.15124) treats this as part of its recipe. Its abstract describes a multi-task evaluation scheme with development evaluations and unseen evaluations, standard benchmark implementations, and decontamination of existing open datasets against those benchmarks. The split is the key idea. You make decisions on the development evals. You report claims on the unseen ones, which no decision touched.
A worked example shows how a small leak makes a fake gain. Suppose the model gets 60% of clean items right, and 95% of items it has seen in training. The new mix leaks 3% of the eval items. The baseline leaks none.
new mix = 0.97 × 60% + 0.03 × 95% = 58.2 + 2.85 = 61.05%. Baseline = 60.00%. Gap = +1.05 points (illustrative)
The gap is about the size of our opening story, and none of it is real. On the clean 97% the two models tie. Here is the same thing as a device.
Both recipes are equally good. Raise the share of eval items that the new mix saw in training, then remove them and compare again.
Illustrative: both recipes get 60% of clean items right, and the model gets 95% of items it saw in training right. The leak is in the new mix only.
You try 16 settings of the learning rate. You score each on the eval and keep the best. Then you report that eval score. You have run Chapter 7's best-of-k: the score is a maximum of 16 noisy numbers, and it is high partly by luck.
The symptom is a result that drops when anyone measures on fresh items. The measurement is to keep one slice of data that no decision ever touches, and to score the final choice once on it. The fix is a three-way split: train to fit, dev to choose among settings, and test to report. Touch the test slice once per claim.
All settings are equally good. Raise the number you try. The first bar is what you would report. The second bar is what fresh items show.
Illustrative: every setting truly scores 60%, and a 2,000 item eval adds luck of 1.1 points per setting (Chapter 3). The chosen bar is the average of 600 imaginary experiments with fixed random numbers.
| Failure | Symptom | Measure | Fix |
|---|---|---|---|
| Confound | Gain arrives with a second change | Config diff; matched-compute rerun | One change per arm |
| Contamination | Gain on one benchmark, none on a sibling | Overlap search, exact then near | Decontaminate; unseen evals |
| Tuned on test | Gain falls on fresh items | Score once on an untouched slice | Train, dev, test split |
Chapter 9
Put the evidence, the error bar and the limits in one small table, so the next reader can check your work in two minutes
A colleague opens your message at the end of a long day. It says "v7 is +0.5 over v6". She has four questions in her head. Is that real? How much did you try? Did the two runs get the same budget? Could I redo it? If the message does not answer them, she will either trust you blindly or ignore the result. Neither is good.
A good write-up answers those questions before they are asked. It is short. It has a table, a sentence about what the data does and does not show, and a link to the run records.
One row per arm. Put the things that make the comparison fair in the same row as the score. Here is a template with made-up numbers.
| Arm | Change | Tokens | Seeds | Score, mean ± SE | Paired gap vs base, 95% CI |
|---|---|---|---|---|---|
| base (v6) | none | 2.0B | 5 | 60.1 ± 1.2 | not applicable |
| mix v7 | data mix only | 2.0B | 5 | 60.6 ± 1.2 | +0.5 [−0.7, +1.7] |
Under the table, write the conclusion in the form of a bounded claim. The numbers above are illustrative:
On 2,000 paired items with 5 seeds per arm and matched tokens, v7 scored +0.5 points over v6 (95% CI −0.7 to +1.7). This design detects gains of about 1.7 points or more.
This sentence is honest in both directions. It does not say "no effect". It says the experiment could not see anything smaller than 1.7 points. A colleague who needs a gain of 1 point now knows this test was too weak, and that the next step is a bigger eval or more seeds, not a decision.
Try the checklist as a reader would. Switch on what the report contains. The meter shows what a colleague can conclude.
Press each item your write-up includes. The meter and the read-out change. Try to reach decision-grade with the fewest items.
This is a teaching scale, not a standard: the labels are ours, and any one missing item can sink a claim. The nine items come from Chapters 2 to 8.
Say five seeds of v7 scored 60.2, 61.0, 60.4, 61.3 and 60.1 on the eval. This is a toy run, so the numbers are illustrative. The mean is the sum, 303.0, divided by 5, which is 60.6.
Next, the spread of the five scores. Their distances from 60.6 are −0.4, 0.4, −0.2, 0.7 and −0.5. Square them: 0.16, 0.16, 0.04, 0.49 and 0.25. The sum is 1.10. Divide by 4 and take the root: 0.52 points.
The standard error of the mean over seeds is 0.52 divided by the square root of 5, which is 0.23 points. That is the number after the plus-or-minus sign in the table, once you add the eval draw from Chapter 3 in quadrature. The failure here is a table that shows only 60.6. Without the spread, the reader cannot tell a quiet result from a lucky one. The measurement is the five raw scores, kept in the manifest. The fix is to print the mean, the error and the seed count together.
Write one manifest per experiment. It lists every run and the facts a stranger needs to redo it.
json{ "experiment": "mix-v7-vs-v6", "primary_metric": "evalset-v3 accuracy, step 3000, paired", "decision_rule": "ship if CI lower bound > 0 and gap >= 1.0", "arms": [ {"name": "base", "config_hash": "a41f", "tokens": 2.0e9, "seeds": [0,1,2,3,4]}, {"name": "mix_v7", "config_hash": "b92c", "tokens": 2.0e9, "seeds": [0,1,2,3,4]} ], "discarded_runs": [{"run": "mix_v7-s5", "reason": "diverged at step 410, reported"}], "contamination_check": "13-gram overlap vs evalset-v3: 0 exact, 4 near matches removed" }
The decision rule is written down before the run. That one line protects you from your own hopes. When the data arrives, you apply the rule, and the argument is short.
Chapter 10
Keep the numbers, the checklist and the code in one place, and find where to go next
One run is a measurement with an error bar you have not drawn. Every habit in this lesson draws that bar, shrinks it, or stops you from reading past it. Change one thing. Match the budget. Count the luck. Compare on the same items. Test proxies before you trust them. Pick the quiet benchmark. Fix the plan before the run. Write the claim with its limits.
| Quantity | Rule or value | Where it came from |
|---|---|---|
| Standard error of a score | √(p(1−p)/n); 1.55 points at p = 0.6, n = 1,000 | Standard statistics; Chapter 3 |
| Standard error of a gap, unpaired | √2 × SE; 2.19 points in that example | Chapter 3 |
| Minimum detectable effect | 2.8 × SE of the gap (80% power, 5% false alarms); 6.1 points | Chapter 3 |
| Items to see a 2 point gain | 9,408 per arm (unpaired, p = 0.6, no training luck) | Chapter 3 |
| Paired standard error | 0.89 points against 2.18 unpaired, same scores | Chapter 4 worked example |
| Best of k seeds | k = 10 reads 1.54 spreads above the truth | Standard statistics; Chapter 7 |
| Chance of a false win | 1 − 0.95m; 64% for 20 ideas | Standard statistics; Chapter 7 |
| Seed sweeps measured | 2,100 BERT fine-tuning trials on four GLUE datasets; initialization and data order matter comparably | Dodge et al., arXiv 2002.06305 |
| Small-scale ranking | 25 corpora; 150M predicts 1B for about 80% of pairs; 3 seeds; up to 100B tokens | DataDecide, arXiv 2504.11393 |
| Cheap proxy metrics | Likelihood metrics make five benchmarks over 80% predictable at 1B for 0.01% of the compute | DataDecide, arXiv 2504.11393 |
| Benchmark signal and noise | 30 benchmarks, 375 models from 60M to 32B parameters | Heineman et al., arXiv 2508.13144 |
pythonimport json, numpy as np def paired_report(path_a, path_b, seeds_a, seeds_b, alpha_z=1.96, power_z=0.84): # per-item correctness keyed by item id, one file per arm load = lambda p: {r["item_id"]: r["correct"] for r in map(json.loads, open(p))} A, B = load(path_a), load(path_b) ids = sorted(A.keys() & B.keys()) d = np.array([B[i] - A[i] for i in ids], dtype=float) # shape (n,) gap, se = d.mean() * 100, d.std(ddof=1) / np.sqrt(len(d)) * 100 # points seed_spread = (np.std(seeds_a, ddof=1) + np.std(seeds_b, ddof=1)) / 2 # training luck, points mde = (alpha_z + power_z) * se return (f"n={len(d)} seeds={len(seeds_a)}+{len(seeds_b)} gap {gap:+.2f} " f"[{gap-alpha_z*se:+.2f}, {gap+alpha_z*se:+.2f}] MDE {mde:.2f} " f"seed spread {seed_spread:.2f}")
Open post-training reports show this discipline in practice. Tulu 3 (arXiv 2411.15124) separates development evals from unseen evals and decontaminates its data against them. OLMo 2 (arXiv 2501.00656) releases its data mixes, code, recipes, logs and thousands of intermediate checkpoints, so others can rerun comparisons and measure noise between checkpoints. Releases like these are what make the checks in this lesson possible for outsiders.
Every paper number in this lesson comes from the abstract page of one of these eight papers. Toy numbers are labelled illustrative where they appear.
Now press Present or Teach and explain, out loud and from memory, how you would test the new data mix: what you would hold fixed, how many seeds and items you would need, and what sentence you would write. Then go back to the experiment at the top, set ten runs per arm, and explain why the better mix now looks better more often.