Controlled Ablations for Post-Training

A new data mix scores 1.1 points higher on one run. Learn to tell a real gain from noise, with seeds, paired tests, power and honest baselines, before you change a post-training recipe.

An ablation is a controlled experiment: change one thing in a training recipe, hold everything else fixed, and measure what moved. Post-training is the stage after pretraining that turns a base model into an assistant.

Press Run the experiment a few times with one run per arm, then switch to ten runs. Count how often the better mix looks worse. Then we build, piece by piece, the habits that stop a lucky run from steering a team.

You need a rough idea of an average and a little Python for the code. We build the statistics from zero.

Built from eight open papers, from Deep Reinforcement Learning that Matters to Signal and Noise. All sources are listed in the last chapter.

Did the new mix help?

Illustrative: the baseline truly scores 60.0% and the new mix 60.5%, a real gain of 0.5 points. Each run adds luck with a spread of 1.5 points. These numbers are made up to show the effect; Chapter 1 gives measured magnitudes from the literature.

Chapter 0

Why One Run Lies

See how a single training run can make a useless change look like a win, and why that matters before a team builds on it

It is Monday. Your team has two versions of a data mix for post-training. Version 6 is the current one. Version 7 adds more code and math examples. You train a model on each and score both on the same set of questions. Version 7 gets 61.4%. Version 6 gets 60.3%.

That is a gain of 1.1 points. Someone says "ship it". Someone else says the next three experiments should build on version 7. Nobody asks the one question that matters: if we ran this again, would version 7 still win?

This lesson is about that question. We will build the habits that answer it, one at a time, with small toys you can play with and the Python you would write.

A few words first

Post-training is the work done after pretraining. Pretraining teaches a model to continue text. Post-training turns it into an assistant, with example conversations, preference data and reward signals. Teams change this recipe all the time: a new data source, a new filter, a different learning rate.

An ablation is an experiment that tests one such change. You build a model with the change and a model without it. You score both. The gap is your evidence. The model without the change is the baseline, also called the control. Each trained model is one run.

To score a run, you give it an eval: a fixed list of questions, called items, each marked right or wrong. The score is the share of items it gets right. Think of a class test. The score is real, but it depends on which questions were on the test and on how the student slept.

Why the gap is not the answer

Training a model has luck in it. The first weights are random. The examples arrive in a shuffled order. A different seed, which is the number that starts the random generator, gives a different run. The recipe is the same, yet the score moves.

So your 1.1 points holds two things mixed together: the true effect of version 7, and the luck of two particular runs. One number cannot split them. A researcher on a frontier post-training team learns early to ask how big the luck is before reading the gap.

The classic warning comes from reinforcement learning. Henderson and colleagues (arXiv 1709.06560) found that results in deep reinforcement learning are hard to reproduce. Nondeterminism in the test environments, plus variance inherent to the methods, made reported gains hard to interpret. They asked for significance metrics and standard reporting. The same warning holds for language models.

The cost of believing luck

Suppose the luck in one run is a spread of 1.5 points. That number is illustrative here; Chapter 1 shows measured spreads. The gap between two single runs then has a spread of about 2.1 points, because both runs bring their own luck.

Now imagine a team that tests twelve ideas, and none of them does anything. Each idea still produces a gap. Some gaps are positive by chance. If the team ships anything above +1 point, a surprising share of the useless ideas passes. Try it below.

Ship twelve useless ideas

Each bar is the measured gap of one idea that truly changes nothing. Red bars pass your shipping bar. Drag the bar up or down, then draw twelve new ideas.

+1.0 points

Illustrative: every idea has a true effect of exactly 0. Each run carries luck with a spread of 1.5 points, so the gap of two single runs has a spread of 2.1 points. The share that passes is computed from that normal curve.

At a bar of +1 point, about one useless idea in three ships. The team then builds the next experiment on top of it. Two more such steps and the recipe holds three changes that do nothing, plus whatever harm they hide. Raising the bar helps. But a higher bar also throws away real gains, and we want a better answer than a guess.

The better answer is to measure the luck, so we know the size of a gap we can trust. That is what the rest of the lesson does. The hero at the top showed the fix in one move: more runs per arm shrink the luck. Chapter 3 makes that exact.

The big idea. A score from one run is a measurement with an error bar you have not drawn yet. An ablation is only as good as your estimate of that error bar.

The road

  1. Where the luck comes from: seed, data order, initialization, eval sampling, nondeterministic kernels.
  2. One change at a time: honest baselines and the matched-compute rule.
  3. How many runs and how big an eval: standard error, minimum detectable effect, a power calculation.
  4. Paired comparisons: scoring both models on the same items.
  5. Small proxies: do rankings from small models hold at scale?
  6. Signal and noise of a benchmark: which evals can support a decision.
  7. Curves, cherry-picks and forking paths: best of k, many comparisons.
  8. Failure modes: confounds, contamination, tuning on the test set.
  9. Writing it up: a table a colleague can trust.
  10. Field guide: numbers, checklist, code, sources.
Version 7 beats version 6 by 1.1 points in one run each. What is the safest reading?

Chapter 1

Where the Luck Comes From

Name the five sources of run-to-run spread, and see how they add up to the error bar on every score

Imagine two cooks who follow one recipe for bread. Both loaves differ a little. The oven runs warm in one corner. One cook kneads a bit longer. The flour came from two different bags. None of this is a different recipe, yet the loaves are not twins.

Training a model is the same. This chapter lists the places where the "same recipe" quietly differs. Once you can name them, you can measure them, and a measured spread is an error bar.

The five sources

Initialization. Some weights start as random numbers. A new seed gives new starting weights, so training walks a different path. In post-training, the base model is usually fixed. Still, new layers, such as a reward head or a LoRA adapter, start at random.

Data order. Training sees examples in batches. The shuffle decides which examples meet in the same batch and which come early. Early and late examples shape the model differently. A new order is a new run.

Sampling during training. Dropout masks, and for reinforcement learning the model's own samples, use random numbers too. Each of them adds spread.

Nondeterministic kernels. A kernel is a small program that runs on the GPU. Many kernels add numbers in an order that changes from launch to launch. Computer arithmetic is not exactly associative, so a different order gives a slightly different sum. Two runs with identical seeds can still drift apart. Our lesson on defeating nondeterminism shows how to remove this one.

Eval sampling. This one is different, because it does not come from training at all. Your eval is a finite list of items. A different list of the same kind would give a different score, even for the same model. We call this the luck of the draw of the questions. Chapter 3 gives its exact size.

What the papers measured

Dodge and colleagues (arXiv 2002.06305) fine-tuned BERT on four GLUE datasets. They ran hundreds of trials per dataset and changed only the random seed. They released the scores of 2,100 trials. Their abstract says that identical settings with distinct seeds can give substantially different results. It also says that weight initialization and data order contribute comparably to the variance.

Madaan and colleagues (arXiv 2406.10229) asked the same question for language model benchmarks. They measured seed variance across initializations, and also how consistent scores are during training. They found that turning a multiple-choice task into a completion task can often lower variance for models near 7B parameters. Standard tools from human testing, such as item response theory, did not reduce it much.

Both papers give one message. The spread is not a footnote. It is large enough to decide which of two recipes looks better. The exact size depends on your task, model and eval, so you must measure it on your own setup. We will show how.

How the sources add up

Independent sources of luck add in a special way. You do not add their sizes. You add their variances. The variance is the square of the spread, where the spread (the standard deviation) is the typical distance of a score from its average. In symbols, for independent sources:

σtotal
The spread of one run's score when every source is active. This is the error bar on a single run.
σinit2
The variance caused by the random starting weights alone.
σorder2
The variance caused by the shuffle of the data alone.
σeval2
The variance caused by which items sit in the eval.

√(0.82 + 0.82 + 1.12 + 0.32) = √(0.64 + 0.64 + 1.20 + 0.09) = √2.57 = 1.6 points (illustrative numbers)

Two things follow. The largest source dominates, because squares grow fast. And removing a small source does little. Try it: switch the sources on and off, and watch 400 imaginary runs of one recipe pile up.

Switch the sources of luck on and off

Each bar counts how many of 400 runs of one recipe landed in that score bin. Press a source to add its luck. Watch the pile widen.

Illustrative spreads in points, not measurements. The eval term is the exact size for 2,000 items at a 60% score (Chapter 3). The 400 runs use fixed random numbers, so switching a source only scales its part.

Notice what the eval term does. Even if you could train with zero luck, you would still see about 1.1 points of spread from the item draw. Only a larger eval, or a paired comparison, removes it. Those are Chapters 3 and 4.

Make the sources visible in your records

You cannot study a source you did not record. Give each source its own seed and write all of them in the run record. Then a seed sweep can vary one at a time.

pythonimport json, random, numpy as np, torch

def set_seeds(init_seed, data_seed):
    random.seed(init_seed); np.random.seed(init_seed)
    torch.manual_seed(init_seed)                       # initialization, dropout
    g = torch.Generator().manual_seed(data_seed)       # data order only
    return g                                           # pass to DataLoader(generator=g)

record = {"run_id": "mix-v7-s2", "recipe": "mix-v7", "init_seed": 2, "data_seed": 2,
          "eval": "evalset-v3", "n_items": 2000, "deterministic_kernels": False,
          "score": 0.612}
# to isolate data order: hold init_seed fixed, sweep data_seed (and the reverse)
print(json.dumps(record))
Three independent sources have spreads of 3, 4 and 0 points. What is the spread of their sum?

Chapter 2

One Change, an Honest Baseline, Matched Compute

Build an experiment where the only difference between two runs is the thing you are testing

A friend says her new running shoes made her faster. You ask what else changed. She also trained twice as often, slept better, and ran on a flat track instead of a hill. Now you cannot say what the shoes did. The shoes were mixed up with other changes.

Researchers call this mixing a confound: a second difference that travels with the first and takes credit for its effect. A controlled ablation exists to remove confounds. It has three rules, and each one is simple to state and easy to break.

Rule 1: change one thing

Take the baseline recipe. Change exactly one field. Keep everything else byte for byte the same: the model, the learning rate, the number of steps, the seeds, the eval. If you want to test two changes, run three more arms: A alone, B alone, and A with B. Then you also learn whether they help each other.

Teams break this rule by accident. A new data mix comes with a new filter, a new template and a new deduplication pass. Each is a change. If you cannot separate them, say so in the write-up, and test them one by one when the result matters.

You can make the rule a test in code. Write each arm as a config, and check that the two configs differ in one field before any GPU time is spent.

pythondef diff(a, b):
    return {k: (a.get(k), b.get(k)) for k in a.keys() | b.keys() if a.get(k) != b.get(k)}

base = {"data_mix": "v6", "lr": 1e-5, "steps": 3000, "batch": 128, "eval": "evalset-v3"}
new  = {"data_mix": "v7", "lr": 1e-5, "steps": 3300, "batch": 128, "eval": "evalset-v3"}
changed = diff(base, new)
assert list(changed) == ["data_mix"], changed   # fails: {'data_mix': ('v6','v7'), 'steps': (3000, 3300)}

The assertion fails here, and that is the point. Someone gave the new run 300 more steps. Without the check, nobody would have noticed.

Rule 2: an honest baseline

The baseline is not a straw man. It must be the best version of the old recipe that you can build with the same effort. If the new recipe got a sweep over four learning rates and the baseline got one, then part of the gap is the sweep, not the idea.

The Deep Reinforcement Learning that Matters paper (arXiv 1709.06560) illustrates how much reported metrics and results vary when you compare against common baselines. That is the lesson here: the baseline's setup is part of the result. Give both arms the same tuning budget, and write the budget down.

Also re-run the baseline in your own pipeline. A number copied from an old table was made with an older eval script, a different seed set, maybe a different tokenizer. It is not the same measurement.

Rule 3: matched compute

Compute is the work a run spends: how many tokens it saw, for how many steps, at what cost in FLOPs (the count of arithmetic operations). More compute usually means a better model, for any recipe. So a run that spends more can win without a better idea.

The fix is to match compute between arms. That means the same number of training tokens and steps. If the new data has longer examples, the same number of examples costs more FLOPs, so you must choose which to match and say which. Often the honest choice is to match FLOPs, and to show the token count next to it.

Here is a worked example with made-up scaling. Suppose each doubling of training tokens adds 2.0 points, and the new mix truly adds 0.4. The new run got 30% more tokens, and the team tuned its learning rate but not the baseline's, worth 0.5 points. The gap is the sum of the parts:

gap = 0.4 (mix) + 2.0 × log2(1.3) (tokens) + 0.5 (tuning) = 0.4 + 0.76 + 0.5 = 1.7 points (illustrative)

Only 0.4 of the 1.7 belongs to the idea. Take the pieces apart yourself.

Where did the gap come from?

The teal part is the data mix. The yellow parts are compute and tuning that leaked in. Give the new run more tokens, then switch on matching and equal tuning.

+30%

Illustrative model, not a measurement: 2.0 points per doubling of tokens, 0.5 points from extra tuning, 0.4 points from the mix. Real scaling is not this tidy, which is why you match compute rather than correct for it.

When both switches are on, the gap shrinks to the true 0.4. You can never read that number off a single comparison with unmatched arms. You can only get it by design.

Open post-training reports such as Tulu 3 (arXiv 2411.15124) and OLMo 2 (arXiv 2501.00656) release their data, code and recipes. That lets others rerun the comparison with matched settings. Treat that as the standard: a result you cannot rerun is a claim, not a measurement.

The new data mix run used 30% more tokens than the baseline and scored 1.2 points higher. What do you do first?

Chapter 3

How Many Runs, How Big an Eval

Turn the luck into a number, the standard error, and use it to plan an experiment that can actually see the effect you care about

A news poll asks 1,000 people whether they like a new law. Sixty percent say yes. The report adds "plus or minus 3". Nobody thinks the true share is exactly 60. The poll saw a small sample, and a different sample would give a slightly different number.

An eval is a poll of questions. Your model answers 1,000 items and gets 60% right. A different set of 1,000 items of the same kind would give another score. The size of that wobble has a formula, and the formula tells you how big your eval must be.

The standard error of a score

Each item is right or wrong. Call the chance of "right" p. The score over n items has a spread, called the standard error (SE): the typical distance between your score and the true score you would see on endless items.

SE
The standard error of the score, as a fraction. Multiply by 100 to get points.
p (1 − p)
How mixed the right and wrong answers are. It peaks at p = 0.5 and shrinks toward 0 or 1.
n
The number of eval items. Four times the items halves the SE.

p = 0.6, n = 1,000: SE = √(0.6 × 0.4 / 1,000) = √0.00024 = 0.0155, which is 1.55 points

Miller (arXiv 2411.00640) argues that eval questions are a draw from an "unseen super-population" of possible questions, and gives formulas for analyzing evals, comparing two models and planning eval experiments. The SE above is the simplest of those formulas. Real evals can have clusters of related items, which make the true SE larger.

The SE of a gap

You compare two models, A and B. Each has its own SE. The gap has a bigger one, because both scores wobble. If A and B are independent, their variances add:

SEgap = √(1.552 + 1.552) = 1.55 × 1.414 = 2.19 points (two 1,000-item scores near 60%)

A 95% confidence interval is the gap plus or minus 1.96 SE. Here that is plus or minus 4.3 points. A measured gap of 2 points on this eval is inside it. The data cannot tell 2 points from zero.

Adding runs

Training luck (Chapter 1) averages out when you train several seeds. The spread of the mean of k runs is the spread of one run divided by the square root of k. Four seeds halve it. But the eval term does not shrink with more seeds, because every seed is scored on the same items. Only more items shrink it.

SEarm = √( p(1 − p)/n + σtrain2/k ), and SEgap = √2 × SEarm

Power and the smallest effect you can see

Before you spend GPU time, ask: if the new mix truly helps by d points, will this experiment notice? The chance that it does is its power. A common target is 80% power with a 5% false alarm rate. For those targets, the minimum detectable effect (MDE) is the true gain you would catch four times out of five:

MDE = (1.96 + 0.84) × SEgap = 2.8 × 2.19 = 6.1 points (1,000 items, one run per arm, no training luck)

Six points! An experiment like that can only see large effects. Most recipe changes add one or two points. To see a 2 point effect with 80% power, solve for n with p = 0.6:

n = 2.82 × 2 × 0.24 / 0.022 = 7.84 × 0.48 / 0.0004 = 9,408 items per arm

That is nearly ten thousand items, and it ignores training luck. The picture below shows the idea. The grey curve is the gap you would measure if nothing changed. The teal curve is the gap you would measure if the true effect were d. The green area is the share of the teal curve past the bar for a false alarm. That share is your power.

Plan the experiment

Choose the eval size, the seeds per arm and the true gain you hope to see. The green area is your power. Make it reach 80%.

1,000
1
+2.0 points

Normal approximation, two-sided 5% false alarm rate, a 60% baseline score, and training luck of 1.2 points per run (illustrative). Items are treated as independent and the two arms as unpaired. Chapter 4 shows how pairing shrinks the spread.

Two sliders matter most. Items fight the eval term; seeds fight the training term. When one term dominates, adding the other does little. Check which term is larger before you buy more of either.

pythonimport numpy as np
from scipy.stats import norm

def se_gap(p, n, sigma_train, k):
    se_arm = np.sqrt(p * (1 - p) / n + sigma_train**2 / k)   # fractions, not points
    return np.sqrt(2) * se_arm

def mde(se, alpha=0.05, power=0.8):
    return (norm.ppf(1 - alpha / 2) + norm.ppf(power)) * se

print(mde(se_gap(0.6, 1000, 0.0, 1)))      # 0.0613 -> 6.1 points
print(mde(se_gap(0.6, 9408, 0.0, 1)))      # 0.0200 -> 2.0 points
Your eval has 1,000 items and your seeds are plentiful, but the gap is still too noisy to read. What most directly lowers the noise?

Chapter 4

Compare Models on the Same Items

Score both models on the same questions and look only at where they disagree, which can cut the error bar by more than half

Two students take the same exam. Student A gets 600 of 1,000 questions right. Student B gets 620. Is B better? Most of the questions were easy, and both got them right. Most of the rest were very hard, and both got them wrong. Those questions say nothing about who is better.

The questions that carry the signal are the ones where the students differ. If B wins nearly all of those, B is better. If the split is close to even, the 20 point lead is luck.

The unpaired mistake

Chapter 3 treated the two scores as independent polls. That ignores a fact: both models answered the same items. Hard items are hard for both. The item draw, which caused much of the error bar, hits both models in the same way, so it largely cancels in the difference.

A paired comparison uses this. For each item, compute the difference between the two models' results. Then treat the list of differences as your data. The item-draw luck that both models share drops out.

A worked example with every step

Take 1,000 items. Model A is right on 600, model B on 620. Sort each item into one of four boxes:

OutcomeItemsDifference B minus A
Both right5700
Only A right30−1
Only B right50+1
Both wrong3500

Check: A is right on 570 + 30 = 600 items, and B on 570 + 50 = 620. The mean difference is (50 − 30) / 1,000 = 0.020, which is 2.0 points. Now the spread of the differences. Each difference is −1, 0 or +1, so the average of the squares is the share of items that disagree, (30 + 50) / 1,000 = 0.080. The variance is that, minus the squared mean:

variance = 0.080 − 0.0202 = 0.0796, so SE = √(0.0796 / 1,000) = 0.00892, which is 0.89 points

The unpaired answer from Chapter 3, for the same scores, is √(0.24/1,000 + 0.2356/1,000) = 0.0218, which is 2.18 points. The paired SE is 2.4 times smaller. In z-scores, the paired gap is 2.0 / 0.89 = 2.24, and the unpaired gap is 2.0 / 2.18 = 0.92. The same data is a clear result with pairing and a shrug without it.

For right-or-wrong items there is a classic test on exactly these counts. McNemar's test looks only at the 30 and the 50. Its statistic is (50 − 30)2 / (50 + 30) = 5.0, a p-value near 0.025. The z-score above is its square root.

SEpaired
The standard error of the gap when both models answer the same items.
disagree
The share of items where exactly one model is right (0.080 in the example). Fewer disagreements mean a smaller error bar.
gap2
The squared mean difference (0.0004 in the example). It is tiny for small gaps.

Now play with it. Keep the two scores fixed at 60% and 62%. Slide the number of disagreeing items. Two models that make similar mistakes disagree on few items, so they pair well.

Move the disagreements

The top bar sorts 1,000 items into four boxes. Below it, two error bars for the same 2.0 point gap. Drag to change how many items only A gets right.

30

Toy counts, with A fixed at 600 right and B at 620 right out of 1,000. B-only items are always 20 more than A-only items. Intervals are the gap plus or minus 1.96 standard errors, from the formulas in the text.

At the far right, the models disagree on almost everything, and pairing is worse than treating them as independent. That is rare in practice. Two models fine-tuned from the same base usually share most of their answers, so pairing usually wins big.

The same idea works when scores are graded instead of right or wrong. Take the per-item score difference, then its mean and standard error. Miller's paper (arXiv 2411.00640) gives formulas for comparing two models and for planning evals in this spirit. Our lessons on eval statistics and error bars on evals go further.

The data flow

This works only if you keep per-item results. Do not save just the final score. Save one record per item, keyed by an item id that is the same for every model.

python# results_A.jsonl / results_B.jsonl: {"item_id": "q0173", "correct": 1}
import json, numpy as np

def load(path):
    return {r["item_id"]: r["correct"] for r in map(json.loads, open(path))}

A, B = load("results_A.jsonl"), load("results_B.jsonl")
ids = sorted(A.keys() & B.keys())                      # same items, same order
d = np.array([B[i] - A[i] for i in ids])               # shape (n,), values -1, 0, +1
gap, se = d.mean(), d.std(ddof=1) / np.sqrt(len(d))
print(f"gap {gap:+.3f}  95% CI [{gap-1.96*se:+.3f}, {gap+1.96*se:+.3f}]")
Two models score 60% and 62% on the same 1,000 items. Which count matters most for how trustworthy the 2 point gap is?

Chapter 5

Small Experiments, Big Decisions

Test whether the winner of a cheap small-scale run is also the winner at full scale, and count how often the ranking flips

A bakery wants to pick the best of 25 cake recipes for a wedding of 1,000 guests. Baking 25 full cakes costs too much. So the baker bakes 25 cupcakes, tastes each, and picks the winner. This works only if a cupcake that tastes best also makes the best big cake.

Frontier teams face this choice all the time. A full-size run costs weeks of a large cluster. A small run costs minutes. If the small run ranks recipes the same way as the big run, you can explore cheaply and spend big compute once. If it does not, you pick the wrong recipe, with confidence.

The DataDecide study

Magnusson and colleagues built a test bed for exactly this question (arXiv 2504.11393). They trained models on 25 different pretraining corpora, which differ in their sources, deduplication and filtering. The models range up to 1B parameters, with up to 100B tokens, and 3 random seeds. They released the models, data and evals.

Their headline finding is simple. Rank the 25 corpora by the score of a model with 150M parameters. That ranking predicts the better corpus at the 1B target scale for about 80% of pairs of corpora. They also tried 8 scaling-law methods, which fit curves across several small sizes. None of those 8 baseline methods beat the compute-decision frontier of single-scale predictions.

One more result matters for evals. Using continuous likelihood metrics at small scale, instead of the usual accuracy, made MMLU, ARC, HellaSwag, MBPP and HumanEval more than 80% predictable at the 1B scale, using just 0.01% of the target compute. We return to why metric choice matters in Chapter 6.

Note the scope. DataDecide studies pretraining data. Post-training recipes, such as the mixes in Tulu 3 (arXiv 2411.15124), may transfer differently. Treat the study as the best open evidence for the method, and check it for your own setting.

Counting how often the ranking holds

The measure is pairwise accuracy. Take every pair of recipes. For each pair, ask: does the small run pick the same winner as the big run? The share of pairs where it does is the pairwise accuracy. With 25 recipes there are 25 × 24 / 2 = 300 pairs, so 80% means 240 correct pairs.

A tiny worked case with three recipes:

RecipeSmall-scale scoreLarge-scale score
A4155
B4058
C3852

The pairs: A against B, small picks A, large picks B, so wrong. A against C, both pick A, right. B against C, both pick B, right. Two of three pairs are right, so the pairwise accuracy is 67%. Notice the wrong pair is the one with the closest small-scale scores, 41 and 40. Close calls flip first.

See why ranks flip

Small runs have their own luck. Each small score is the true quality plus noise. When the noise is larger than the gap between two recipes, their order is a coin flip. Below, 25 imaginary recipes have a true large-scale quality. The left column is their small-scale order and the right column is their large-scale order. A red line is a recipe that moved five places or more.

Do small rankings hold at scale?

Each line joins one recipe's rank at small scale to its rank at large scale. Crossing lines are pairs the small run got wrong. Raise the noise, or switch to a quieter metric.

2.0 points

Toy model, not DataDecide's data: 25 recipes whose true large-scale scores are spread over about 12 points, plus Gaussian noise at small scale. DataDecide measured about 80% at 150M parameters for 1B, from its 25 corpora. The quieter metric halves the noise for illustration only.

The accuracy depends on the noise, but also on the gaps. Recipes that differ by a lot are almost never swapped. Recipes that differ by a hair often are. That is fine if you only need to find a clearly better recipe. It is a problem if you want to rank a crowd of near-equals.

A practical rule follows. Use small runs to throw away clear losers and to shortlist a few. Then confirm the finalists at the scale you care about, with the seeds and paired checks from Chapters 3 and 4. And before you trust the method, run two or three recipes at both scales and see whether their order agrees.

pythonfrom itertools import combinations

def pairwise_acc(small, large):          # dicts: recipe -> score
    pairs = list(combinations(small, 2))
    right = sum((small[a] - small[b]) * (large[a] - large[b]) > 0 for a, b in pairs)
    return right / len(pairs)

small = {"A": 41, "B": 40, "C": 38}
large = {"A": 55, "B": 58, "C": 52}
print(pairwise_acc(small, large))          # 0.667
Which pair of recipes is the small-scale run most likely to rank in the wrong order at large scale?

Chapter 6

Signal and Noise in a Benchmark

Measure how well an eval separates good models from bad, and how much it wobbles on its own, before you let it make a decision

You step on a bathroom scale. It reads 70.2 kg. You step on again. It reads 73.1 kg. Then 68.9 kg. The scale wobbles by about 2 kg every time. Could you notice that you gained half a kilo this week? No. The scale is too noisy for that question.

A benchmark is a scale for models. Some benchmarks are steady. Others jump around even when the model barely changes. Before you use one to settle an ablation, you should know which kind it is.

Two numbers: signal and noise

Heineman and colleagues (arXiv 2508.13144) give names to the two numbers. Signal is how well a benchmark separates better models from worse ones. Noise is how sensitive the score is to random variability between training steps. They studied 30 benchmarks and 375 open-weight models, from 60M to 32B parameters, and released about 900K evaluation results.

Here is a plain version you can compute. Take a set of models that really differ. The signal is the spread of their scores. Then take one model and read its score at the last several checkpoints, which are saved copies of the model from nearby training steps. The noise is the spread of those scores. The ratio is the signal-to-noise ratio, SNR.

SNR
How many noise-widths separate the models you want to tell apart. Higher is better.
σmodels
The spread of final scores across different models. This is the signal.
σcheckpoints
The spread of one model's scores over its late checkpoints. This is the noise.

This is a simplified form of the paper's idea, written so you can compute it in ten lines. Read the paper for its exact definitions. The key claim holds either way: the paper argues for benchmarks that are more reliable for small-scale decisions, and it tests ways to raise signal or cut noise.

A worked example

Take six models scoring 44, 46, 48, 51, 53 and 56 on a benchmark. The mean is 49.7. The squared distances from it are 32.1, 13.5, 2.8, 1.8, 11.1 and 40.1. Their sum is 101.4. Divide by 5 and take the root: the signal is 4.5 points.

Benchmark X wobbles by 0.9 points between checkpoints. Its SNR is 4.5 / 0.9 = 5. Benchmark Y wobbles by 3.0 points, so its SNR is 1.5. Now ask how often each benchmark puts two models 2 points apart in the right order. The score of each model has noise σ, so the difference has noise √2 × σ. The chance of the right order is the normal curve at 2 / (1.414 × σ):

X: Φ(2 / (1.414 × 0.9)) = Φ(1.57) = 94%. Y: Φ(2 / (1.414 × 3.0)) = Φ(0.47) = 68%

On the steady benchmark you get the order right about 19 times in 20. On the noisy one, about two times in three. Same models, same gap, a different tool.

Three ways to buy back signal

The paper tests three interventions. Each one is cheap.

  1. Switch the metric. Perplexity-style scores, which use the probability the model gives to the right answer, are smoother than accuracy, which flips from right to wrong in one jump. The paper tests this kind of switch, for example perplexity rather than accuracy, as one of its three interventions.
  2. Filter noisy subtasks. A suite is often an average of many parts. Dropping the parts that are mostly noise raises the SNR of the average.
  3. Average checkpoints. Average the scores of the last several checkpoints. If their noise is independent, averaging k of them shrinks it by the square root of k. The paper tests averaging the output of a model's intermediate checkpoints as a way to reduce noise.

Madaan and colleagues found a related effect (arXiv 2406.10229): turning a multiple-choice task into a completion task often reduces variance for models around 7B. Both papers say the same thing in different words. Choose the measurement that is quiet before you choose the model that is loud.

Can this benchmark tell them apart?

Each column is one model. Each dot is one reading of its score. When columns overlap, the benchmark cannot order those models. Switch benchmark, average checkpoints, or smooth the metric.

1

Toy numbers: six models at 44, 46, 48, 51, 53 and 56, so the signal is 4.5 points. Benchmark noise is 0.9 or 3.0 points per reading. Averaging k checkpoints assumes independent noise, which real checkpoints only partly satisfy. The 0.6 factor is an illustration, not a measured value.

pythonimport numpy as np

# final[m]: score of model m.  last[m]: its scores at the last 10 checkpoints.
signal = np.std(list(final.values()), ddof=1)                       # spread across models
noise  = np.mean([np.std(last[m], ddof=1) for m in last])   # wobble within a model
snr = signal / noise
smooth = {m: np.mean(last[m][-5:]) for m in last}              # average last 5 checkpoints
Two benchmarks separate your models equally well, but benchmark Y has three times the checkpoint-to-checkpoint noise. What is true of its SNR?

Chapter 7

Curves, Cherry-Picks and Forking Paths

See how choosing the best run, the best checkpoint or the best metric after the fact builds a gain out of nothing

A student takes a test ten times and hands in the best score. Another student takes it once. Whose score tells you more about what they know? The first student's score is a maximum, and a maximum of ten noisy tries is almost always higher than a typical try. It measures luck as much as skill.

Ablations fall into this trap quietly. The trap has three forms: pick the best seed, pick the best checkpoint, and test many ideas and report the winner. All three give a gain that disappears when someone reruns the experiment.

Reading a training curve

A training curve plots a score against training steps. A single curve per arm hides the luck. Plot every seed as its own thin line, and the mean as a thick one. Then ask three plain questions.

  1. Do the arms separate, and stay apart? A gap that appears at one step and vanishes at the next is noise.
  2. Do the curves cross? If they cross, the answer depends on when you stop, so the stopping step must be fixed before you look.
  3. Do some seeds diverge? Dodge and colleagues (arXiv 2002.06305) report that on small datasets many fine-tuning trials diverge partway through. They offer guidance on stopping the less promising runs early.

Stopping early is fine when a rule you set in advance does it, using a dev set (data held out for decisions) and not the final test. It is a cherry-pick when you stop at the step where your favorite arm looks best.

Best of k

Dodge and colleagues also measured how the best-found performance changes with the number of trials. More trials give a higher maximum, even though the recipe did not change. Take a recipe with run-to-run spread σ. The expected best of k runs sits above the true score by a multiple of σ:

Runs, kExpected best, in spreads above the truthWith σ = 1.5 points
10.00+0.0
51.16+1.7
101.54+2.3
1002.51+3.8

These are standard facts about the maximum of normal draws. They are not from the papers. They show the size of the effect: with ten tries and a spread of 1.5 points, a recipe that does nothing looks 2.3 points better. Try it.

Report the best of k runs

Every run here is the same recipe with a true score of 60. Grey shows single runs. Yellow shows the best of k runs. Raise k.

10

Illustrative: run-to-run spread of 1.5 points, 600 imaginary experiments with fixed random numbers. Checkpoints of one run are correlated, so picking the best checkpoint inflates less than picking the best of independent seeds, but the direction is the same.

The cure is a rule: decide the number of seeds and the stopping step before you run. Report the mean and the spread across all seeds, never the best one. If you ran twelve and show eight, say why.

The garden of forking paths

Now widen the view. Suppose you test 20 ideas that all do nothing. Each test has a 5% false alarm rate. The chance that none raises a false alarm is 0.95 to the power 20, which is 0.36. So with probability 64% you will see at least one "significant" idea. This is the multiple comparisons problem.

P(at least one false alarm) = 1 − 0.95m; for m = 20: 1 − 0.358 = 64%

The problem is bigger than it looks, because the 20 are often hidden. You looked at three metrics, four eval subsets, two checkpoints and three seed sets. That is 72 ways to cut the same data, and you only reported the cut that looked good. Statisticians call this the garden of forking paths. Each fork was a reasonable choice. Together they are a search.

Two fixes work together. First, write the analysis plan before the run: one primary metric, one checkpoint rule, one seed set, one decision threshold. Everything else is labeled exploratory. Second, when you must test many things, tighten the bar. The simplest tightening is Bonferroni's rule: use 0.05 / m as the bar for each test. It is blunt, and it keeps the total false alarm rate near 5%.

Test many ideas that do nothing

Each dot is the p-value of one idea that truly does nothing. A p-value is the chance of seeing a gap this big by luck alone. Red dots fall under the bar and look like wins. Press the button for a new batch.

20

Every idea has a true effect of exactly zero, so each p-value is a uniform random number between 0 and 1. The bar is 0.05, or 0.05 divided by m with Bonferroni. The chance of at least one false alarm is computed from these rules.

You trained 10 seeds of one recipe and report only the best seed as "the result". What does the number mean?

Chapter 8

Three Ways an Ablation Goes Wrong

Spot confounded changes, leaked eval items and a test set that was tuned on, each by its symptom, its measurement and its fix

A teacher gives a class a practice sheet. Later, the real exam has the same questions. The class scores very high. Did they learn the subject, or did they memorize the sheet? From the score alone, you cannot tell.

Ablations can fail in three ways that look like success. For each, the symptom is a gain that seems too good, or that vanishes on a rerun. Then you need a measurement that separates the real from the fake, and a fix.

Failure 1: confounded changes

Chapter 2 covered this one. The symptom is a gain that comes with a second change: more tokens, more tuning, a new filter, a new chat template. The measurement is a config diff that lists every field that differs, plus a re-run with matched compute. The fix is to change one thing, or to add arms that isolate each change.

Failure 2: eval contamination

Contamination means that eval items, or close copies of them, appear in the training data. The model then remembers answers, and the score measures memory, not skill. Post-training mixes are built from many open datasets, and some of those contain benchmark questions.

The symptom is a gain on one benchmark with no gain on a different one that tests the same skill. The measurement is to count the overlap. Search the training data for each eval item: exact string matches first, then near matches, such as the same question with different spacing or a paraphrase. The fix is decontamination: remove the matching training examples, or drop the matching eval items from the comparison.

Tulu 3 (arXiv 2411.15124) treats this as part of its recipe. Its abstract describes a multi-task evaluation scheme with development evaluations and unseen evaluations, standard benchmark implementations, and decontamination of existing open datasets against those benchmarks. The split is the key idea. You make decisions on the development evals. You report claims on the unseen ones, which no decision touched.

A worked example shows how a small leak makes a fake gain. Suppose the model gets 60% of clean items right, and 95% of items it has seen in training. The new mix leaks 3% of the eval items. The baseline leaks none.

new mix = 0.97 × 60% + 0.03 × 95% = 58.2 + 2.85 = 61.05%. Baseline = 60.00%. Gap = +1.05 points (illustrative)

The gap is about the size of our opening story, and none of it is real. On the clean 97% the two models tie. Here is the same thing as a device.

Leak some eval items into training

Both recipes are equally good. Raise the share of eval items that the new mix saw in training, then remove them and compare again.

3.0%

Illustrative: both recipes get 60% of clean items right, and the model gets 95% of items it saw in training right. The leak is in the new mix only.

Failure 3: tuning on the test set

You try 16 settings of the learning rate. You score each on the eval and keep the best. Then you report that eval score. You have run Chapter 7's best-of-k: the score is a maximum of 16 noisy numbers, and it is high partly by luck.

The symptom is a result that drops when anyone measures on fresh items. The measurement is to keep one slice of data that no decision ever touches, and to score the final choice once on it. The fix is a three-way split: train to fit, dev to choose among settings, and test to report. Touch the test slice once per claim.

Pick the best setting on the test set

All settings are equally good. Raise the number you try. The first bar is what you would report. The second bar is what fresh items show.

10

Illustrative: every setting truly scores 60%, and a 2,000 item eval adds luck of 1.1 points per setting (Chapter 3). The chosen bar is the average of 600 imaginary experiments with fixed random numbers.

FailureSymptomMeasureFix
ConfoundGain arrives with a second changeConfig diff; matched-compute rerunOne change per arm
ContaminationGain on one benchmark, none on a siblingOverlap search, exact then nearDecontaminate; unseen evals
Tuned on testGain falls on fresh itemsScore once on an untouched sliceTrain, dev, test split
A new data mix gains 2 points on benchmark A and nothing on benchmark B, which tests the same skill. What do you check first?

Chapter 9

Write It Up So a Colleague Trusts It

Put the evidence, the error bar and the limits in one small table, so the next reader can check your work in two minutes

A colleague opens your message at the end of a long day. It says "v7 is +0.5 over v6". She has four questions in her head. Is that real? How much did you try? Did the two runs get the same budget? Could I redo it? If the message does not answer them, she will either trust you blindly or ignore the result. Neither is good.

A good write-up answers those questions before they are asked. It is short. It has a table, a sentence about what the data does and does not show, and a link to the run records.

The table

One row per arm. Put the things that make the comparison fair in the same row as the score. Here is a template with made-up numbers.

ArmChangeTokensSeedsScore, mean ± SEPaired gap vs base, 95% CI
base (v6)none2.0B560.1 ± 1.2not applicable
mix v7data mix only2.0B560.6 ± 1.2+0.5 [−0.7, +1.7]

Under the table, write the conclusion in the form of a bounded claim. The numbers above are illustrative:

On 2,000 paired items with 5 seeds per arm and matched tokens, v7 scored +0.5 points over v6 (95% CI −0.7 to +1.7). This design detects gains of about 1.7 points or more.

This sentence is honest in both directions. It does not say "no effect". It says the experiment could not see anything smaller than 1.7 points. A colleague who needs a gain of 1 point now knows this test was too weak, and that the next step is a bigger eval or more seeds, not a decision.

What must be in the report

  1. The single change, as a config diff.
  2. The baseline, re-run in the same pipeline with the same tuning budget.
  3. The budget: tokens, steps and, if they differ, FLOPs.
  4. The seeds: how many, and which ones, for each arm.
  5. The eval: its name and version, the number of items, and the standard error.
  6. The paired gap with its confidence interval, and the minimum detectable effect.
  7. A contamination check: what you searched, and what you found.
  8. The plan: the primary metric and the stopping rule, written before the run.
  9. Every run you started, including failed and discarded ones, with the reason.

Try the checklist as a reader would. Switch on what the report contains. The meter shows what a colleague can conclude.

What can a colleague conclude?

Press each item your write-up includes. The meter and the read-out change. Try to reach decision-grade with the fewest items.

This is a teaching scale, not a standard: the labels are ours, and any one missing item can sink a claim. The nine items come from Chapters 2 to 8.

Fill in one row, step by step

Say five seeds of v7 scored 60.2, 61.0, 60.4, 61.3 and 60.1 on the eval. This is a toy run, so the numbers are illustrative. The mean is the sum, 303.0, divided by 5, which is 60.6.

Next, the spread of the five scores. Their distances from 60.6 are −0.4, 0.4, −0.2, 0.7 and −0.5. Square them: 0.16, 0.16, 0.04, 0.49 and 0.25. The sum is 1.10. Divide by 4 and take the root: 0.52 points.

The standard error of the mean over seeds is 0.52 divided by the square root of 5, which is 0.23 points. That is the number after the plus-or-minus sign in the table, once you add the eval draw from Chapter 3 in quadrature. The failure here is a table that shows only 60.6. Without the spread, the reader cannot tell a quiet result from a lucky one. The measurement is the five raw scores, kept in the manifest. The fix is to print the mean, the error and the seed count together.

Keep the records that make it checkable

Write one manifest per experiment. It lists every run and the facts a stranger needs to redo it.

json{
  "experiment": "mix-v7-vs-v6",
  "primary_metric": "evalset-v3 accuracy, step 3000, paired",
  "decision_rule": "ship if CI lower bound > 0 and gap >= 1.0",
  "arms": [
    {"name": "base", "config_hash": "a41f", "tokens": 2.0e9, "seeds": [0,1,2,3,4]},
    {"name": "mix_v7", "config_hash": "b92c", "tokens": 2.0e9, "seeds": [0,1,2,3,4]}
  ],
  "discarded_runs": [{"run": "mix_v7-s5", "reason": "diverged at step 410, reported"}],
  "contamination_check": "13-gram overlap vs evalset-v3: 0 exact, 4 near matches removed"
}

The decision rule is written down before the run. That one line protects you from your own hopes. When the data arrives, you apply the rule, and the argument is short.

An experiment's paired gap is +0.5 points with a 95% interval from −0.7 to +1.7. What is the most honest sentence for the report?

Chapter 10

Field Guide

Keep the numbers, the checklist and the code in one place, and find where to go next

One run is a measurement with an error bar you have not drawn. Every habit in this lesson draws that bar, shrinks it, or stops you from reading past it. Change one thing. Match the budget. Count the luck. Compare on the same items. Test proxies before you trust them. Pick the quiet benchmark. Fix the plan before the run. Write the claim with its limits.

The numbers and terms

QuantityRule or valueWhere it came from
Standard error of a score√(p(1−p)/n); 1.55 points at p = 0.6, n = 1,000Standard statistics; Chapter 3
Standard error of a gap, unpaired√2 × SE; 2.19 points in that exampleChapter 3
Minimum detectable effect2.8 × SE of the gap (80% power, 5% false alarms); 6.1 pointsChapter 3
Items to see a 2 point gain9,408 per arm (unpaired, p = 0.6, no training luck)Chapter 3
Paired standard error0.89 points against 2.18 unpaired, same scoresChapter 4 worked example
Best of k seedsk = 10 reads 1.54 spreads above the truthStandard statistics; Chapter 7
Chance of a false win1 − 0.95m; 64% for 20 ideasStandard statistics; Chapter 7
Seed sweeps measured2,100 BERT fine-tuning trials on four GLUE datasets; initialization and data order matter comparablyDodge et al., arXiv 2002.06305
Small-scale ranking25 corpora; 150M predicts 1B for about 80% of pairs; 3 seeds; up to 100B tokensDataDecide, arXiv 2504.11393
Cheap proxy metricsLikelihood metrics make five benchmarks over 80% predictable at 1B for 0.01% of the computeDataDecide, arXiv 2504.11393
Benchmark signal and noise30 benchmarks, 375 models from 60M to 32B parametersHeineman et al., arXiv 2508.13144

The checklist, one line each

  1. Before: one changed field, a config diff, a baseline re-run with equal tuning, matched tokens and FLOPs.
  2. Plan: primary metric, stopping rule, seeds, decision threshold, all written first. Compute the MDE and check that it is below the gain you care about.
  3. During: separate seeds for init and data order. Keep per-item results, keyed by item id.
  4. After: mean and standard error over seeds, a paired gap with its interval, a contamination check, every run listed.
  5. Claim: a sentence with the gap, the interval and the smallest gain the design could see.

The core as code

pythonimport json, numpy as np

def paired_report(path_a, path_b, seeds_a, seeds_b, alpha_z=1.96, power_z=0.84):
    # per-item correctness keyed by item id, one file per arm
    load = lambda p: {r["item_id"]: r["correct"] for r in map(json.loads, open(p))}
    A, B = load(path_a), load(path_b)
    ids = sorted(A.keys() & B.keys())
    d = np.array([B[i] - A[i] for i in ids], dtype=float)       # shape (n,)
    gap, se = d.mean() * 100, d.std(ddof=1) / np.sqrt(len(d)) * 100  # points
    seed_spread = (np.std(seeds_a, ddof=1) + np.std(seeds_b, ddof=1)) / 2   # training luck, points
    mde = (alpha_z + power_z) * se
    return (f"n={len(d)} seeds={len(seeds_a)}+{len(seeds_b)}  gap {gap:+.2f} "
            f"[{gap-alpha_z*se:+.2f}, {gap+alpha_z*se:+.2f}]  MDE {mde:.2f}  "
            f"seed spread {seed_spread:.2f}")

Where it sits in the field

Open post-training reports show this discipline in practice. Tulu 3 (arXiv 2411.15124) separates development evals from unseen evals and decontaminates its data against them. OLMo 2 (arXiv 2501.00656) releases its data mixes, code, recipes, logs and thousands of intermediate checkpoints, so others can rerun comparisons and measure noise between checkpoints. Releases like these are what make the checks in this lesson possible for outsiders.

Keep going

Sources

Every paper number in this lesson comes from the abstract page of one of these eight papers. Toy numbers are labelled illustrative where they appear.

  1. Henderson et al. Deep Reinforcement Learning that Matters. arXiv 1709.06560
  2. Dodge et al. Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping. arXiv 2002.06305
  3. Madaan et al. Quantifying Variance in Evaluation Benchmarks. arXiv 2406.10229
  4. Miller. Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations. arXiv 2411.00640
  5. Magnusson et al. DataDecide: How to Predict Best Pretraining Data with Small Experiments. arXiv 2504.11393
  6. Heineman et al. Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluation. arXiv 2508.13144
  7. Lambert et al. Tulu 3: Pushing Frontiers in Open Language Model Post-Training. arXiv 2411.15124
  8. Team OLMo et al. 2 OLMo 2 Furious. arXiv 2501.00656
The takeaway. A gain is a number plus a reason to believe it. The reason is a design: one change, matched budget, enough seeds and items, a paired comparison, a plan fixed in advance, and a claim that states what the experiment could not see.

Now press Present or Teach and explain, out loud and from memory, how you would test the new data mix: what you would hold fixed, how many seeds and items you would need, and what sentence you would write. Then go back to the experiment at the top, set ten runs per arm, and explain why the better mix now looks better more often.

Controlled Ablations for Post-Training
Henderson et al. · Dodge et al. · Madaan et al. · Miller · DataDecide · Signal and Noise · Tulu 3 · OLMo 2 · Back to Gleams