Model A beats model B by 2 points on a benchmark. Before you believe it, audit the test itself: its size, its labels, its leaks, its ceiling and its ranking.
Learn to read a benchmark score the way an auditor does: what it measures, how much noise sits in it, and what you may safely claim from it.
Drag the number of test items up and down, then add label errors. Watch the gap between the two models grow, shrink, or sink into noise. Then we build, piece by piece, every check an audit needs.
You need the idea of an average and a little Python for the code. We build the rest from zero.
Built from seven papers: label errors in test sets, MMLU errors, GSM1k, a contamination survey, benchmark agreement testing, the Leaderboard Illusion and HELM. Every number from a paper is cited; toy numbers say so.
Illustrative: model A truly answers 84% of items right and model B 82%. A wrong label turns a right answer into a miss, and a wrong answer into a hit one time in three (four-choice items). The whiskers are the standard 95% interval for two independent scores.
Chapter 0
Follow two models, one benchmark and a 2 point lead, and learn why a score is an estimate and not a fact
Priya trains a new model, B. Her team already uses model A. Both run on one benchmark: 500 questions, each with one right answer. A gets 420 right, which is 84.0%. B gets 410 right, which is 82.0%. The weekly report says "A is better". A manager asks whether to keep A.
Priya asks a different question. If we wrote 500 new questions of the same kind, would A still win? The answer decides whether the 2 point gap is a fact about the models or a fact about those 500 questions.
A benchmark is a fixed set of test questions with an answer key. Each question is an item. A model's accuracy is the share of items it answers right. That share is an average over a sample. It only estimates what you want to know: how often the model would be right on all questions of this kind, including the ones nobody wrote.
Think of tasting one spoon of soup. The spoon tells you about the pot, but a different spoon tastes a little different. The spread between spoons is sampling error. A small test is a small spoon.
Each item is a coin flip with a fixed chance p of coming up "right". For n items, the spread of the average is the standard error: the typical distance between one spoon and the pot.
Work it for Priya's models. For A, p = 0.84, so SE = √(0.84 × 0.16 / 500) = √0.000269 = 0.0164, or 1.64 points. For B, p = 0.82, so SE = √(0.82 × 0.18 / 500) = √0.000295 = 0.0172, or 1.72 points.
SE of the gap = √(0.0164² + 0.0172²) = √0.000564 = 0.0238, or 2.38 points
The gap is 2.0 points and its own error is 2.38 points. The gap is smaller than one standard error. A 95% confidence interval is the range that would catch the true value in 95 of 100 repeated tests. For a gap it is the gap plus or minus 1.96 standard errors: 2.0 ± 4.7, which runs from −2.7 to +6.7. Zero sits inside. The test cannot tell A from B.
Try the hero at the top of the page now. Drag the items to 500 and read the interval it prints. Then drag to 10,000 and watch the whisker shrink until zero falls outside.
The formula says one in five new tests would put B ahead. You can check that by brute force. The device below writes a fresh 500-item test again and again. Each time, A and B answer it, and the device records the gap.
Each bar counts test sets by the gap they showed, in whole points. Warm bars: A ahead. Red bars: B ahead. Press Draw 200 and compare the red share with the 20% the formula predicts.
Illustrative: both models answer each of 500 fresh items independently, A right 84% of the time and B right 82%. Real models share hard items, which narrows the spread, so the real chance of a flip is smaller than this toy shows. The draws are seeded, so you see the same sequence on every visit.
The pile of gaps looks like a bell, centred near +2. About one fifth of it lies to the left of zero. In those test sets, a team would have written "B is better" and kept B. Same models. Same skill. A different spoon of questions.
The sampling story covers only one way a score misleads. A benchmark can also measure the wrong thing. It can hold wrong answers in its key. It can leak into training, run out of room at the top, or rank models differently from the next benchmark. An audit asks about each, in this order.
Each chapter ends with something you can run. By the last chapter you will write an audit report from a real sample of items, with an error rate, a noise floor and a recommended use. If you want the statistics behind intervals first, the eval statistics lesson builds them from zero, and AI evaluation covers how evals are built.
pythonimport numpy as np def gap_interval(correct_a, correct_b): # correct_a, correct_b: 0/1 arrays of shape (n,), one entry per item n = len(correct_a) pa, pb = correct_a.mean(), correct_b.mean() se = np.sqrt(pa*(1-pa)/n + pb*(1-pb)/n) # independent-scores version gap = pa - pb return gap, gap - 1.96*se, gap + 1.96*se # 500 items: A right on 420, B right on 410 (the story's numbers) a = np.r_[np.ones(420), np.zeros(80)] b = np.r_[np.ones(410), np.zeros(90)] print(gap_interval(a, b)) # (0.02, -0.027, 0.067): zero is inside
Chapter 1
Separate the skill you intend to test from the cues a model can use instead, and measure the difference
A teacher gives a maths test with four-choice questions. One student never reads a question. She notices that the right answer is the longest one, and she picks it every time. She scores 55%. Another student solves every problem and scores 94%. The test says the first student knows 55% of the maths. She knows none of it.
The score is real. It just does not measure what the teacher meant it to. Benchmarks suffer from the same problem, and the problem has a name.
A construct is the ability you want to know about but cannot see directly: "reasoning", "reading comprehension", "writes correct code". You cannot weigh a construct. You build a test, run it, and read a number. Construct validity asks one question: does that number move when the ability moves, and only then?
Write the gap as two behaviours. The intended behaviour is what the benchmark's name promises, such as solving a multi-step problem. The measured behaviour is whatever actually earns points on the items. When the two differ, the score answers a different question from the one on the cover.
Four causes show up again and again. The table lists them. Each row is a general pattern, so the examples are illustrative.
| The benchmark claims | What can earn the points instead | How to check |
|---|---|---|
| Reasoning | A cue in the options, such as the longest answer or a repeated word | Hide the question and score on options alone |
| Reading comprehension | Facts the model already knows, so the passage is never needed | Hide the passage and score again |
| Instruction following | Output format alone, such as ending with the right keyword | Read 30 passing answers by hand |
| Knowledge | A recall of a memorised test question | Chapters 3 and 4: contamination checks |
Notice the pattern. Each check removes the intended skill and asks whether the score survives. If it does, the skill was not what the score measured. This is the cheapest and most useful test in the whole lesson.
Suppose a share s of the items can be answered by a cue alone. A model that ignores the questions gets all of those right, and guesses on the rest. With four choices, a guess is right one time in four. A model with the real skill gets the cue items right and most of the rest.
The grid is the benchmark: blue cells are items that need the skill, yellow cells give the answer away. The bars compare a model that never reads the question with a model that has the skill. Drag the share of giveaway items.
Illustrative: four-choice items; the no-skill model guesses at 25% on items without a cue; the skilled model is right on 90% of them. The giveaway share is the knob; no real benchmark is drawn here.
At 40% giveaway items, the no-skill model scores 0.40 + 0.60 × 0.25 = 55.0%. The skilled model scores 0.40 + 0.60 × 0.90 = 94.0%. The gap between them is 39 points. With no giveaways at all, the same two models differ by 65 points. The cues did two harms at once. They inflated the weak model by 30 points above chance and squeezed the range in which skill can show.
The choice-only baseline hides the question and shows only the options. If a model beats chance by more than sampling error allows, the options leak the answer. You already know the error formula from Chapter 0.
pythonimport numpy as np def choice_only_check(items, ask_model, n_choices=4): # items: list of {"question": str, "options": [str]*4, "answer": int} hits = [] for it in items: prompt = "Pick the most likely correct option.\n" + "\n".join(it["options"]) # no question hits.append(ask_model(prompt) == it["answer"]) acc, n, chance = np.mean(hits), len(hits), 1/n_choices se = np.sqrt(chance*(1-chance)/n) return acc, chance, (acc - chance)/se # z above 2 means the options leak
A z-score is the distance from chance counted in standard errors. A z above 2 is hard to blame on luck. In the story, 0.55 on 500 items gives z = (0.55 − 0.25) / 0.0194 = 15.5, which is not close.
A second route to validity is to refuse a single number. The HELM project (arXiv 2211.09110) evaluated 30 prominent language models across 42 scenarios and 7 metrics: accuracy, calibration, robustness, fairness, bias, toxicity and efficiency. Before HELM, the authors report, models were evaluated on only 17.9% of its 16 core scenarios on average. After, 96.0%. The point for an auditor is simple: a benchmark that reports only accuracy has one view of a model, and construct validity improves when several views agree.
Write down for any benchmark you audit: the intended behaviour in one sentence, the cheapest check that removes it, and the score the check leaves behind. If the leftover score is high, report it next to the headline.
Chapter 2
Estimate how many labels are wrong from a small re-checked sample, then correct the score you measured
An auditor picks 100 items from a benchmark at random. She reads each question, works out the answer herself, and compares it with the answer key. Seven keys are wrong. One says a river flows into the wrong sea. One has two right options and marks only one. A third has a typo that makes every option false.
Every model that gets those seven items "wrong" may have been right. The benchmark has taught the grader to punish the correct answer.
A label is the answer key's entry for an item. A label error is an entry that is wrong, or that does not have one clear right answer. The share of items with such an entry is the label error rate, written e.
The rate sets a ceiling. A model that is perfect on every item still loses points on the items whose key is wrong, so the best score anyone can reach is about 1 − e. We call that the noise floor of the benchmark: the point where the remaining "mistakes" are the key's mistakes, not the model's. If the top score is 93.5% and the noise floor is 6.5%, then two models near 93% are separated by noise.
More common than most people assume. The paper "Pervasive Label Errors in Test Sets" (arXiv 2103.14749) studied 10 widely used datasets across vision, language and audio. It estimates an average of at least 3.3% label errors across their test sets, and at least 6% in the ImageNet validation set. It also shows why this changes decisions. On corrected ImageNet, ResNet-18 beats ResNet-50 once mislabeled test items rise by just 6%. On CIFAR-10, VGG-11 beats VGG-19 at a 5% rise. Noise can reverse the ordering of models.
Language benchmarks have the same disease. The paper "Are We Done with MMLU?" (arXiv 2406.04127) estimates that 6.49% of MMLU questions contain errors. In the Virology subset, 57% of the questions the authors analysed contain errors. The authors re-annotated 5,700 questions across all 57 subjects and released them as MMLU-Redux. They report that using it changes the performance numbers that had been reported before.
You cannot re-check every item. You can re-check a random sample and use the same standard error as in Chapter 0, now for the error rate. Take n items at random, have careful people re-annotate them, and count the wrong keys, k.
The auditor found k = 7 in n = 100. So p̂ = 0.07, the standard error is √(0.07 × 0.93 / 100) = 0.0255, and the interval is 7.0% ± 5.0%, which is 2.0% to 12.0%. That is wide. With n = 400 and k = 26, p̂ = 6.5%, the standard error is 1.23 points, and the interval is 6.5% ± 2.4%, which is 4.1% to 8.9%. Four times the checking halves the width, as the square root promised.
Each cell is one re-checked item: red means the key was wrong. The bar below shows your estimate and its interval. Choose a sample size, press Check a new sample, then Reveal the true rate.
Illustrative: a benchmark of 2,000 items in which 6.5% of keys are wrong (close to the MMLU estimate, but a made-up benchmark). Each press draws a new random sample, seeded so the sequence repeats. The interval uses the normal approximation, which is rough when k is small.
Now use the rate. When a key is wrong, a model that answers correctly gets marked down. A model that answers wrongly is marked up only if its wrong answer happens to match the wrong key, which on a four-choice item is about one time in three. Put those two effects together and the measured score m relates to the true accuracy p like this.
m = p × (1 − 4e/3) + e/3 so p = (m − e/3) / (1 − 4e/3)
Take m = 82.0% and e = 6.5%. Then e/3 = 0.0217 and 1 − 4e/3 = 0.9133. So p = (0.820 − 0.0217) / 0.9133 = 0.7983 / 0.9133 = 0.874. The model's true accuracy is near 87.4%, not 82.0%. The key hid 5.4 points of skill. The same factor shrinks every gap between models by 0.9133, so a 2.0 point gap would be 2.2 points on a clean key.
Do not apply this blindly. It assumes the wrong keys are scattered at random and that the model is right where the key is wrong. In practice, items with wrong keys are often ambiguous, and a strong model may disagree with the corrected answer too. Treat the corrected number as a bound that shows the size of the effect, and report both.
Checking at random wastes effort on good items. A model can point at likely suspects first. In confident learning, you collect each item's predicted probabilities from a model that did not train on that item. For every class you find the model's average confidence on items that carry that label. That average is the class's bar. An item becomes a suspect when the model clears the bar for a different class than the key says. A person then checks the suspect. Example: an item is labelled "cat", the model gives "dog" 0.92, and the bar for "dog" is 0.70. The item is a suspect.
pythonimport numpy as np def label_error_estimate(n_checked, n_wrong): p = n_wrong / n_checked h = 1.96 * np.sqrt(p*(1-p)/n_checked) return p, max(0, p-h), p+h def correct_accuracy(measured, e, n_choices=4): q = 1/(n_choices-1) # chance a wrong answer matches a wrong key return (measured - e*q) / (1 - e*(1+q)) print(label_error_estimate(100, 7)) # (0.07, 0.020, 0.120) print(correct_accuracy(0.82, 0.065)) # 0.874
The failure mode, as symptom then measurement then fix. Symptom: the top models on a benchmark sit in a tight cluster and "mistakes" on the hardest items look odd. Measurement: re-annotate a random sample and read the disputed items. Fix: report the error rate with its interval, publish a corrected subset when you can, and stop ranking models whose gap is smaller than the noise floor.
Chapter 3
See how test items leak into training data, and run the checks that catch a leak
A student sits an exam. The night before, a friend slipped her a sheet with the exact questions and answers. She scores 96%. Her teacher concludes that she understands the course. The teacher is measuring memory of a sheet.
Language models can have the same sheet. They learn from huge piles of web text, and benchmarks live on the web.
Benchmark data contamination is what happens when a model's training data holds information from the benchmark it is later tested on. The survey "Benchmark Data Contamination of Large Language Models" (arXiv 2406.04244) defines it this way: models "inadvertently incorporate evaluation benchmark information from their training data", which makes the results inaccurate or unreliable.
It is rarely a plot. It is usually an accident, and it has a few common routes:
Notice that nobody has to cheat. A lab that never opens the test file can still train on its echoes. That is why the fix is measurement, not trust.
If you can read the training data, the plain test is to look for the item in it. An n-gram is a run of n words in a row. "gives 12 to a food bank" holds the 4-grams "gives 12 to a", "12 to a food" and "to a food bank". For one test item, count how many of its n-grams also appear in the training corpus. The share is the overlap. Flag the item when the share passes a line you set, say 50%.
Two choices matter, and the device lets you feel both. If n is too small, ordinary phrases match and clean items look contaminated. If n is too large, a light edit breaks every long sequence and a leaked item looks clean.
Pick a version of the test item. Yellow words belong to at least one n-gram found in the training text. The curve shows how the overlap falls as n grows; the dashed line is the 50% flag line.
Illustrative: a made-up bakery word problem and a short made-up corpus that holds the original item, its worked answer and some neighbouring text. The overlap is computed live in your browser on lower-cased words with punctuation removed.
At n = 4 the device flags the copy (100%) and the light edit (63%) and passes the clean item (43%). At n = 1 it flags nearly everything, because common words match. At n = 8 it misses the light edit (38%). The paraphrase scores 3% at n = 2 and 0% beyond. Word-sequence overlap cannot see a paraphrase, because the words changed even though the meaning did not.
Often you cannot read the training data. Then you probe the model. Two probes are common.
The perplexity probe compares how surprised a model is by the original item against how surprised it is by paraphrases of the same item. Perplexity is a score of surprise: lower means the model found the text more predictable. If a model finds the original wording clearly less surprising than equal-quality rewrites, it has probably seen that wording. Suppose the average loss per token is 0.9 on the original and 1.6 on five paraphrases (illustrative numbers). That gap is a warning sign. A single item proves nothing, so run it over a whole benchmark and compare the average gap with the gap on items you know are fresh.
The completion probe gives the model the first half of an item and checks whether it writes the second half word for word. A model that has not seen the item almost never reproduces a specific second half of 20 words.
pythondef ngrams(words, n): return {" ".join(words[i:i+n]) for i in range(len(words)-n+1)} def overlap(item, corpus_ngrams, n=4): g = ngrams(item.lower().split(), n) return len(g & corpus_ngrams) / max(1, len(g)) # share of the item's n-grams found def perplexity_gap(model_loss, original, paraphrases): # model_loss(text) -> mean negative log-likelihood per token return sum(model_loss(p) for p in paraphrases)/len(paraphrases) - model_loss(original) # a clearly positive gap, averaged over many items, suggests the wording was seen
Each check has a blind spot. Overlap needs the corpus and misses paraphrases. The perplexity probe is noisy per item and fooled by items that are simply easy to predict. The completion probe misses a model that learned the answer but not the wording. For this reason, no single check settles the question. They move your belief up or down. The only direct test is one that the model could not have seen, which is the next chapter.
Chapter 4
Build a fresh parallel test, measure the gap on it, and learn what that gap can and cannot prove
A teacher suspects that last year's exam leaked. She cannot search every student's notes. So she writes a new exam: same topics, same style, same difficulty, different questions. Students who learned the subject score about the same on both. Students who memorised the leak drop.
That is the cleanest test for contamination, and researchers have run it on a famous maths benchmark.
A fresh parallel set is a new test, written to match an old one in format, topic and difficulty, with no item copied. You score each model on both. The contamination gap for a model is its old-test accuracy minus its fresh-test accuracy. If a model really has the skill, the gap is near zero. If it has memorised the old items, it scores high on them and drops on the fresh ones.
The paper "A Careful Examination of Large Language Model Performance on Grade School Arithmetic" (arXiv 2405.00332) did exactly this. The authors built GSM1k, a new set of grade-school maths problems made to mirror GSM8k, a widely used benchmark. They report accuracy drops of up to 8%, and evidence of systematic overfitting in several families of models across almost all model sizes. They also report a positive link, with Spearman r² = 0.36, between how likely a model is to generate an example from GSM8k and the size of its drop on GSM1k. The more a model finds GSM8k examples natural to write, the more it fell. That fits the idea that some models partly memorised the old set. The authors also note that many frontier models showed little sign of overfitting, so the drop is not a rule for every model.
The gap is a difference of two averages, so it has the error bar from Chapter 0. Take a model that scores 78.0% on 1,000 old items and 72.0% on 1,000 fresh items (illustrative numbers). The gap is 6.0 points. The standard error is √(0.78 × 0.22 / 1000 + 0.72 × 0.28 / 1000) = √(0.0001716 + 0.0002016) = 0.0193, or 1.93 points. The 95% interval is 6.0 ± 3.8, which is 2.2 to 9.8. Zero is outside. This model dropped for real.
Now a second model: 70.0% old, 69.0% fresh. The gap is 1.0 point and the standard error is √(0.00021 + 0.000214) = 2.06 points. The interval is 1.0 ± 4.0, from −3.0 to +5.0. Zero is inside. You have no evidence of a leak for this model, and no proof of its absence either.
Each dot is a toy model. Across: how familiar the old test items look to it. Up: how far its score fell on the fresh test. Drag the memorisation strength and the measurement noise, and watch the link appear and blur.
Illustrative: 24 invented models. A model's drop is its familiarity (0 to 1) times the memorisation strength, plus random noise. The paper’s own figures are real: drops of up to 8%, and a Spearman r² of 0.36 between a model’s chance of generating a GSM8k example and its gap on GSM1k.
Set memorisation to 0 and the dots form a flat cloud: no link, whatever the noise. Raise it and the cloud tilts. With strong memorisation and little noise, the link is close to a line. With strong noise, the same memorisation hides in scatter, and the r² falls toward the paper’s middling 0.36.
A gap proves a leak only if the fresh set is a fair twin. Check four things before you read the number.
pythonimport numpy as np def contamination_gap(old_correct, fresh_correct): # two 0/1 arrays, one entry per item, from different item sets po, pf = old_correct.mean(), fresh_correct.mean() se = np.sqrt(po*(1-po)/len(old_correct) + pf*(1-pf)/len(fresh_correct)) gap = po - pf return gap, gap - 1.96*se, gap + 1.96*se # model X: 780/1000 old, 720/1000 fresh (illustrative) x_old = np.r_[np.ones(780), np.zeros(220)]; x_new = np.r_[np.ones(720), np.zeros(280)] print(contamination_gap(x_old, x_new)) # (0.06, 0.022, 0.098)
A gap is a symptom. Its causes include memorisation, but also a harder twin, a format the model handles worse, and plain luck. To tell them apart, line up the gap with the other checks. If the same models also show a perplexity gap (Chapter 3) and a big familiarity effect, memorisation is the best explanation. If only the gap appears, suspect difficulty.
The fix has three layers. Report the old and the fresh score side by side. Prefer the fresh score for ranking. Retire an old benchmark once the gap and the leak evidence are both large. The survey in Chapter 3 also discusses assessment designs that reduce the risk of leaks, which is the long-term direction.
Chapter 5
Find the point where a benchmark stops separating strong models, and decide what it is still good for
A leaderboard lists six models. Their scores read 85.0, 90.0, 94.0, 97.0, 98.5 and 99.5. The lowest models are far apart. The top three are crowded into two and a half points. A new model arrives at 99.0, and a blog post calls it "state of the art".
Is the top of this list a ranking of skill, or a ranking of luck? The answer depends on how much room is left on the scale.
A benchmark is saturated when the best models score so near the maximum that more skill no longer shows up as a higher score. The maximum is the ceiling. A ceiling effect is what you see when scores pile up against it: the scale has run out of room, so different abilities all read the same.
Chapter 2 gave you the real ceiling. With a share e of wrong keys, even a perfect model reads only 1 − e. If e = 3%, the visible ceiling is 97.0%, not 100%. A model at 97% is at the noise floor, and any score it reads above that is luck in the key. Remember the formula from Chapter 2 for what a model with true accuracy p will read: m = p × (1 − 4e/3) + e/3.
First, the key runs out. Once the models' remaining mistakes are mostly the key's mistakes, extra skill cannot raise the score. Second, a difference needs enough items to see. To detect a gap d between two models with accuracy near p, you need about this many items per model.
n ≈ 2 × 1.96² × p(1 − p) / d²
Work an example. Two models near p = 0.97 differ by d = 0.005, half a point. Then n = 2 × 3.84 × 0.0291 / 0.000025 = 8,944 items. A benchmark of 1,000 items, which is common, is nine times too small for that question. At a gap of one point it needs about 2,240 items. This matters because the models you most want to compare are exactly the ones whose gaps are smallest.
Each bar is a model’s measured score with its 95% interval. The dashed line is the ceiling, 1 minus the label noise. A green link means the test can tell two neighbours apart. Drag the label noise and the number of items.
Illustrative: six models with true accuracy 85.0, 90.0, 94.0, 97.0, 98.5 and 99.5. Measured scores follow m = p(1 − 4e/3) + e/3 for four-choice items. A pair counts as separable when the 95% interval of the gap excludes zero (independent scores).
At 3% label noise and 1,000 items, the top bar reads 96.5% under a ceiling at 97.0%. Only the pairs in the lower half are separable. Raise the items to 20,000 and more pairs turn green, but the ceiling does not move. More items buy resolution, not headroom. To move the ceiling you must fix the key.
A second view helps near the top. Compare error rates. A move from 97% to 98% is a score change of one point. It is also a drop from 3 errors in 100 to 2 in 100, which is a third fewer errors. A drop from 85% to 86% removes about 1 error in 15. The same one point means different things at different heights, which is why a leaderboard difference of "one point" is not one fixed amount of skill. But the error view has the same flaw: when the key's own mistakes make up most of the errors, a cut in "errors" is a cut in nothing.
It is not worthless. It changes job.
pythonimport math def items_needed(p, d, z=1.96): # items per model to detect an accuracy gap d near accuracy p return math.ceil(2 * z*z * p*(1-p) / (d*d)) def visible_ceiling(label_error): return 1 - label_error # best score a perfect model can read print(items_needed(0.97, 0.005)) # 8944 print(items_needed(0.97, 0.01)) # 2236 print(visible_ceiling(0.03)) # 0.97
The failure mode, as symptom then measurement then fix. Symptom: new models gain a tenth of a point at a time and every release is "best". Measurement: compute the ceiling from the label error rate, then count how many items you would need for the gap you see. Fix: if the ceiling is within a few points of the top score, or the item count is far below the number needed, call the top of the list a tie.
Chapter 6
Compare how two benchmarks order the same models, and see which choices change the answer
A team launches a new benchmark for coding agents. To show it is sound, they run eight models on it and on an older, trusted benchmark. The two lists look alike. The authors write: "our benchmark agrees with the old one". A reader asks the obvious question. Agrees how much, and on which models?
Agreement between benchmarks is one of the oldest ways to check a new test. It is also easy to do badly.
Scores on two benchmarks have different units, so we compare orders instead. A model's rank is its place in the list: 1 for the best. Spearman rank correlation, written ρ, measures how well two rank lists match. It is 1 when the lists are identical, 0 when they are unrelated, and −1 when one is the reverse of the other.
Work six models. On benchmark A their ranks are 1, 2, 3, 4, 5, 6. On benchmark B they are 2, 1, 3, 5, 4, 6. The differences are 1, 1, 0, 1, 1, 0, so Σ d² = 4. Then ρ = 1 − 6 × 4 / (6 × 35) = 1 − 0.114 = 0.886. Two small swaps among neighbours leave a high correlation.
The paper "Benchmark Agreement Testing Done Right" (arXiv 2407.13696) points out that benchmark agreement testing (BAT) has no standard procedure. The authors analysed over 40 prominent benchmarks and found that some overlooked methodological choices can significantly change BAT results. They propose a set of best practices and release a Python package, BenchBench, together with a leaderboard that rates benchmarks against their peers.
You can see one such choice in the device below. Two benchmarks test partly overlapping skills. Move the share of skill they have in common, and then change which models you include in the comparison.
Left column: the order on the trusted benchmark. Right column: the order on the new one. Lines that cross show models that swap places. Drag the shared skill, then choose which models count.
Illustrative: eight invented models with two skills each. The trusted benchmark scores skill 1. The new one scores a mix of skill 1 (the shared part) and skill 2. Ranks inside the chosen subset are recomputed, and ρ is the formula above.
With 60% shared skill, the two benchmarks agree at ρ = 0.93 over all eight models. Pick only the top four and the same two benchmarks agree at 0.60. Only the bottom four: 0.80. Nothing about the benchmarks changed. The choice of models did. A tight cluster of strong models has small gaps, and small gaps reorder easily. So the headline "ρ = 0.93" hides a fact: the benchmarks disagree about exactly the models you most want to separate.
pythonfrom scipy.stats import spearmanr, kendalltau def agreement(scores_a, scores_b, subset=None): # scores_a, scores_b: dict model -> score on each benchmark models = subset or sorted(scores_a) a = [scores_a[m] for m in models]; b = [scores_b[m] for m in models] return {"n": len(models), "spearman": spearmanr(a, b)[0], "kendall": kendalltau(a, b)[0]} # always report the full set AND the top slice top4 = sorted(scores_a, key=scores_a.get, reverse=True)[:4] print(agreement(scores_a, scores_b), agreement(scores_a, scores_b, top4))
Noise explains part of this. Take two models near 90% on 1,000 items whose true gap is 1 point. Each score has a standard error of √(0.9 × 0.1 / 1000) = 0.95 points, so the gap has one of 1.34 points. The chance that noise puts the weaker model ahead is the chance of a draw 1 / 1.34 = 0.75 standard errors below the mean, which is about 23%. Nearly one pair in four swaps on a retest. A second benchmark adds its own noise, so two lists that measure the same skill still disagree about close neighbours.
High agreement means the benchmarks order models alike. It does not mean either is valid. Two benchmarks can agree because both reward the same shortcut (Chapter 1), both leaked into the same training sets (Chapter 3), or both are saturated in the same way (Chapter 5). Read the result with a decision table:
| Agreement | For a new benchmark that claims a new skill | For a benchmark meant to replace an old one |
|---|---|---|
| High | Suspicious: it may measure the old skill again | Good: rankings carry over |
| Middling | Plausible: related but different | Check the top end before you switch |
| Near zero | Check for noise or breakage first | Do not switch |
Chapter 7
See how picking the best of many tries inflates a published score, and what a leaderboard must disclose to be trusted
A lab trains 27 versions of a model. Each version is a little different: a new data mix, a different learning rate, a changed system prompt. The lab tests all 27 on a public leaderboard before release. It publishes the score of the best one. The other 26 scores are never shown.
The published score is true for that one version on that test. It is not the score you should expect from the next version, and it is not a fair comparison with a lab that tested one model. Something has been selected, and the selection is invisible.
Every score has noise (Chapter 0). If you take several noisy scores and keep the highest, you keep the noise that happened to point up. This is the winner's curse: the best of many is, on average, luckier than it is better. Think of ten people each flipping 100 fair coins. The top flipper may report 62 heads. The coin did not get better. The top slot went to the luckiest hand.
For N variants of equal true strength, with score noise of size σ, the expected best score sits above the truth by about σ times a growth factor that rises slowly with N. For a bell-shaped noise the factor is about 0.56 for 2 tries, 1.16 for 5, 1.54 for 10 and 2.0 for 27. Drag N in the device and watch the published score climb even though no variant is better than another.
All variants are truly equal. The curve shows how far the best score of N tries sits above the truth. The dashed tick marks 27 variants. Drag N and the score noise.
Illustrative: scores on an Elo-style scale with bell-shaped noise, simulated live over 4,000 seeded trials for each N. The paper reports that one provider tested 27 private variants before a release; it does not give the size of the inflation, so none is quoted from it.
With noise of 15 points and 27 variants, the best score sits about 30 points above the truth. Test one variant and the inflation is zero. The size of the effect depends on how many tries were hidden, which is exactly the number a reader cannot see.
The paper "The Leaderboard Illusion" (arXiv 2504.20879) studies the Chatbot Arena, a leaderboard where people vote between pairs of model answers. It reports four things an auditor should know.
The second and fourth findings are different problems. Selection is about which scores are shown. Data access is about who gets to train and tune against the test's own style. Both bend the leaderboard away from what it claims to measure, which is how good a model is for a person asking a question.
Ask six plain questions. Each has a measurable answer.
pythonimport numpy as np def expected_inflation(n_variants, sigma, trials=4000, seed=0): # all variants equally good; each score = truth + N(0, sigma) noise rng = np.random.default_rng(seed) draws = rng.normal(0, sigma, size=(trials, n_variants)) return draws.max(axis=1).mean() # mean of the published (best) score for n in (1, 5, 27): print(n, round(expected_inflation(n, 15), 1)) # about 0, 17, 30 points
Sometimes you have only the published number. You can still estimate its luck. Say a lab reports 1,030 on an Elo-style scale after testing 27 variants, and the score noise is 15 points. The expected inflation is about 30 points, so a fair estimate of that model's strength is near 1,000. This is a rough fix, because the real noise and the real N are guesses. It still beats taking the headline at face value. If a lab will not say N, widen the guess.
Disclose N, and publish every score, including retractions. Report an interval, and treat overlapping intervals as a tie. When you must pick the best variant, test it again on fresh items that played no part in the choice. The fresh score carries no luck from the selection, and it is the number to quote. A lab that does this loses a few points of headline and gains a number that holds up.
Chapter 8
Turn a sample of items, an error taxonomy and four checks into one page that says how a score may be used
Priya has now run the checks. She holds a notebook full of numbers: wrong keys, a choice-only score, an overlap count, a ceiling. Her manager does not want the notebook. The manager wants to know one thing. "Can I put this number in a decision?"
An audit report answers that question in one page. It lists what was checked, what was found, and what the score may and may not be used for. The last line matters most. It is a permission, not a score.
Every audit report has the same seven parts. The order follows the order of this lesson.
| Part | What goes in it | Where we built it |
|---|---|---|
| 1. Scope | Benchmark name and version, number of items, who wrote it, the claim it makes | Chapter 1 |
| 2. Item sample | How many items were re-checked, how they were drawn (random, seeded), who checked them | Chapter 2 |
| 3. Error taxonomy | Counts per kind of problem: wrong key, ambiguous, several right answers, giveaway cue, out of scope | Chapters 1 and 2 |
| 4. Noise floor | The estimated error rate with its interval, and the ceiling 1 minus that rate | Chapters 2 and 5 |
| 5. Contamination check | Overlap result, probe result, fresh-set gap if one exists, and what each cannot rule out | Chapters 3 and 4 |
| 6. Agreement | Rank correlation with named benchmarks, for all models and the top slice | Chapter 6 |
| 7. Recommended use | Allowed uses, banned uses, and a date to re-audit | This chapter |
Draw the item sample with a recorded random seed, after the benchmark is frozen. Write the seed and the list of item numbers in the report, so another auditor can draw the same items. Think about the groups inside the benchmark too. MMLU-Redux spans all 57 MMLU subjects, so a random sample of 100 items gives fewer than two items per subject. A claim about one subject needs a sample drawn for that subject.
A taxonomy is a list of named kinds, so that two auditors count the same way. Keep it short and give each kind a test. A wrong key is an item whose listed answer is false. An ambiguous item has two defensible answers, or a question that can be read two ways. A giveaway can be answered without the skill (Chapter 1). An out-of-scope item tests something the benchmark does not claim to test. Count each kind separately, because each leads to a different fix: repair the key, rewrite the item, drop the item, or narrow the claim.
The last part is a decision. Use rules you can state before you see the results, so the outcome cannot drift to fit a wish. Here is a simple set. It is this lesson's illustration, not a standard.
Set what your audit found. The report below rewrites itself, and its last line gives the permission. Try a small sample first, then a large one.
The thresholds (2 points, 5 points, 5%, 15%) are the lesson’s illustrative house rules, not a published standard. Problem items are wrong keys plus ambiguous items. The interval uses the normal approximation from Chapter 2.
With the starting values, 14 problem items in 200 give a 7.0% problem rate, with an interval of 3.5% to 10.5%. The ceiling is 93.0%. The best score of 91.0% leaves 2.0 points of headroom, so the rule says: tiers, not ranks. Move the best score to 94% and headroom becomes negative. The score is above the ceiling, which means the key penalises correct answers or the model has seen the items.
A report is more useful when a program can read it. Store it as a record next to the benchmark, and re-run the audit when the benchmark changes.
pythonimport json, math def audit_report(name, n, wrong, ambiguous, leak_share, best): bad = (wrong + ambiguous) / n h = 1.96 * math.sqrt(bad*(1-bad)/n) ceiling = 1 - bad headroom = ceiling - best if leak_share > 0.15: use = "upper bound only; plan to replace" elif headroom < 0.02: use = "regression test only" elif headroom < 0.05 or bad > 0.05: use = "tiers, not ranks" else: use = "ranking, with intervals" return {"benchmark": name, "items_checked": n, "problem_rate": round(bad, 3), "interval": [round(max(0, bad-h), 3), round(bad+h, 3)], "ceiling": round(ceiling, 3), "headroom": round(headroom, 3), "recommended_use": use} print(json.dumps(audit_report("toy-bench", 200, 10, 4, 0.04, 0.91), indent=1))
The symptom of a missing report is a number that travels without its caveats: a slide, a table, a model card. The measurement is the checklist above. The fix is to attach the one-page report to every score you share, and to put the recommended use on the same slide as the number.
Chapter 9
Measure the auditor, plant known errors, and learn the ways an audit goes stale, biased or gamed
Priya's audit says 5.5% of the keys are wrong. A colleague runs it again with different readers and finds 8%. Which audit is right? Neither, exactly. Both are measurements, and both have error. An audit that no one audits is one more number to trust on faith.
This chapter lists the main ways audits fail, and shows how to measure each one.
An auditor is like a smoke alarm. Recall is the share of real fires it sounds for. The false alarm rate is the share of times it sounds with no fire. A human reader re-checking keys has both. A tired reader misses subtle errors. An eager reader flags good items as bad.
If the true error rate is e, an auditor with recall r and false alarm rate f reports this rate.
reported = e × r + (1 − e) × f so e = (reported − f) / (r − f)
Take e = 6.5%, r = 70% and f = 1%. Then reported = 0.065 × 0.70 + 0.935 × 0.01 = 0.0455 + 0.0094 = 5.5%. This matches Priya's number. The truth is higher than her audit says, because her readers missed three in ten real errors.
The benchmark truly has 6.5% wrong keys. The bars show what your auditor reports, and what you get after correcting with measured recall and false alarms. Drag the auditor’s recall and false alarm rate.
Illustrative: the 6.5% is the made-up benchmark of Chapter 2. The corrected bar is exact only when recall and false alarm rate are known, which is why the next section plants known errors to measure them.
Set recall to 100% and false alarms to 0 and the report equals the truth. Lower the recall and the report sinks below the truth. Raise the false alarms and it floats above. The two errors can cancel by accident, which is the worst case: the audit looks right and nothing is known.
Decide the width you want before you start. For an error rate near p, the half-width of the interval is 1.96 √(p(1 − p)/n), so n = 1.96² p(1 − p) / h². For p = 6.5% and a half-width of 2 points, n = 3.84 × 0.0608 / 0.0004 = 584 items. For a half-width of 1 point you need about 2,335. Auditors who stop at 100 items and report a point estimate are reporting a number with a 5 point margin.
To measure an auditor, give them a mixed batch. Add items that you know have wrong keys (copy a few items and corrupt the key), and items you know are clean. Do not tell the auditor which are which. The share of planted errors they catch is their recall. The share of known-clean items they flag is their false alarm rate. Then correct the headline number with the formula above. This is the same trick a test engineer uses when injecting known bugs to check a test suite.
A second reader gives another view. Cohen's kappa measures agreement beyond what luck would give. Two auditors check 100 items. Both flag 8. Auditor 1 alone flags 4, and auditor 2 alone flags 3. Both leave 85 alone. Observed agreement is (8 + 85) / 100 = 0.93. By luck, they would agree on a fraction 0.12 × 0.11 + 0.88 × 0.89 = 0.0132 + 0.7832 = 0.7964. Kappa is (0.93 − 0.7964) / (1 − 0.7964) = 0.66. That is decent, not perfect. The grader reliability lesson goes deeper into measuring raters.
pythondef corrected_rate(reported, recall, false_alarm): return (reported - false_alarm) / (recall - false_alarm) def cohens_kappa(both, only1, only2, neither): n = both + only1 + only2 + neither po = (both + neither) / n p1, p2 = (both + only1) / n, (both + only2) / n pe = p1*p2 + (1-p1)*(1-p2) return (po - pe) / (1 - pe) print(corrected_rate(0.055, 0.70, 0.01)) # about 0.065 print(cohens_kappa(8, 4, 3, 85)) # about 0.66
Each failure has a symptom, a measurement and a fix, like the ones in earlier chapters. Anchoring: symptom, readers rarely disagree with the key. Measurement: compare answer-first and key-first readers on the same items. Fix: answer first.
Chapter 10
Keep the numbers, the formulas, the checklist and the code, and know where to read next
A benchmark score is an estimate with noise, read through an answer key that may be wrong, from a test the model may have seen, on a scale that may have no room left. An audit checks each of those in turn. It starts with the interval on the gap. Then it asks what the items measure and how many keys are wrong. It asks whether the items leaked, how much headroom remains, whether other benchmarks agree, and who chose the number. It ends with a permission: the uses for which the score is safe.
| Finding | Number | Source |
|---|---|---|
| Average label errors across 10 test sets (at least) | 3.3% | Label errors, arXiv 2103.14749 |
| Label errors in the ImageNet validation set (at least) | 6% | Label errors, arXiv 2103.14749 |
| MMLU questions with errors (estimate) | 6.49% | MMLU, arXiv 2406.04127 |
| Virology questions with errors, of those analysed | 57% | MMLU, arXiv 2406.04127 |
| Questions re-annotated for MMLU-Redux | 5,700 | MMLU, arXiv 2406.04127 |
| Largest accuracy drop on the fresh GSM1k set | up to 8% | GSM1k, arXiv 2405.00332 |
| Spearman r², familiarity with GSM8k against the gap | 0.36 | GSM1k, arXiv 2405.00332 |
| Prominent benchmarks analysed for agreement testing | over 40 | Agreement, arXiv 2407.13696 |
| Private variants tested by one provider before a release | 27 | Leaderboard Illusion, arXiv 2504.20879 |
| Estimated Arena data share: Google, OpenAI, 83 open-weight models together | 19.2%, 20.4%, 29.7% | Leaderboard Illusion, arXiv 2504.20879 |
| HELM: scenarios, metrics, models | 42, 7, 30 | HELM, arXiv 2211.09110 |
| HELM: average core-scenario coverage before and after | 17.9% to 96.0% | HELM, arXiv 2211.09110 |
| Question | Formula | Chapter |
|---|---|---|
| Error of a score | SE = √(p(1 − p)/n); interval = ± 1.96 SE | 0 |
| Ceiling from label noise | ceiling = 1 − e | 2, 5 |
| True accuracy from a measured score | p = (m − e/3) / (1 − 4e/3), four choices | 2 |
| Items to detect a gap d | n ≈ 2 × 1.96² × p(1 − p) / d² | 5 |
| Rank agreement | ρ = 1 − 6 Σ d² / (n(n² − 1)) | 6 |
| Auditor-corrected rate | e = (reported − f) / (r − f) | 9 |
pythonimport numpy as np def audit(correct_a, correct_b, recheck, choice_only, overlap, best): # correct_a, correct_b : 0/1 arrays (n,) per-item results of two models # recheck : 0/1 array (m,) 1 = auditor found the key wrong or ambiguous # choice_only : 0/1 array (n,) a model that never saw the question # overlap : array (n,) share of each item's n-grams found in training text n = len(correct_a); pa, pb = correct_a.mean(), correct_b.mean() se = np.sqrt(pa*(1-pa)/n + pb*(1-pb)/n) # Ch 0: gap interval gap = (pa - pb, pa - pb - 1.96*se, pa - pb + 1.96*se) e = recheck.mean(); ceiling = 1 - e # Ch 2: noise floor z_cue = (choice_only.mean() - 0.25) / np.sqrt(0.25*0.75/n) # Ch 1: cue leakage leak = (overlap >= 0.5).mean() # Ch 3: flagged share headroom = ceiling - best # Ch 5 use = ("replace" if leak > 0.15 else "regression only" if headroom < 0.02 else "tiers" if headroom < 0.05 or e > 0.05 else "rank with intervals") return dict(gap=gap, error_rate=e, ceiling=ceiling, cue_z=z_cue, leak=leak, headroom=headroom, use=use)
Auditing sits between building an evaluation and acting on it. AI evaluation shows how evals are built and run. Eval statistics gives the interval maths in depth. Grader reliability measures the raters who score open-ended answers, the same problem as Chapter 9 for model outputs. Post-training ablations uses benchmark scores to pick between training recipes, so every number it relies on should pass this audit first. Embedding benchmarks applies the same questions to retrieval tests.
Now press Present or Teach and explain one audit out loud, from memory: the interval, the sample of keys, the leak check, the ceiling, the ranking check and the permission you would write. Then go back to the two models at the top, set the items to 10,000 and the wrong labels to 8%, and explain why the gap is real but smaller.