Grader Reliability

A grader is a measuring instrument, and instruments can lie. Change only the grader and the same two models can swap places in the ranking.

Learn how to measure a judge before you trust its scores: its biases, its agreement with people, its mistakes, and what they do to a win rate.

Pick a grader, then press Run the eval again a few times. Model A is truly better. Watch which grader says so. Then we build, piece by piece, how to test, correct and ship a grader you can trust.

You need a feel for percentages and a little Python for the code. We build the rest from zero.

Built from eight primary sources, each opened and checked: MT-Bench and Chatbot Arena, Wang et al. on unfair evaluators, G-Eval, Length-Controlled AlpacaEval, JudgeBench, Gao et al. on reward model overoptimization, Shankar et al. on validating validators, and Miller on error bars. Links in the last chapter. All numbers inside the devices are labelled illustrative unless a paper owns them.

Same models, four graders

Illustrative, not from a paper. Model A truly passes 60% of 400 tasks and model B 55%. Each grader passes a truly good answer with its sensitivity and a truly bad one with 1 minus its specificity: Noisy 85% and 85%, Lenient 95% and 70%, Style-fond 90% and 80% for A but 95% and 55% for B, whose longer answers it likes. Whiskers show a 95% binomial interval. Every run draws fresh random verdicts in your browser.

Chapter 0

A Grader Is a Measuring Instrument

See why every score from a judge model is a reading on an instrument, and learn the four ways an instrument goes wrong

Your team changes a prompt. You run the eval, a set of 300 test questions, and a judge model grades the answers. The pass rate moves from 50% to 58%. Everyone cheers.

A week later a teammate runs the same answers through the same judge again. This time the pass rate is 52%. Nothing about the model changed. The judge provider quietly updated the judge, and its opinion moved by six points.

Which number was real? You cannot tell. That is the whole problem of this lesson. A score is only as good as the thing that produced it, and a model that grades other models is a thing that can be wrong in quiet ways.

The plain idea: a grader is a bathroom scale

Think of a bathroom scale. You step on it each morning and trust the number. You trust it because the scale has been right before, not because the display looks serious. A scale can fail, and it can fail in different ways.

A grader is anything that turns a model’s answer into a score. It can be a rule (does the output contain the right number?), a checklist called a rubric, a person, or another language model. A language model used this way is called an LLM judge. The whole set of questions, answers and graders together is an eval, short for evaluation. The headline number it reports, such as the share of questions passed, is a pass rate. When two models are compared answer by answer, the share of comparisons one wins is its win rate.

Every grader is a measuring instrument. So we can borrow the words engineers use for scales, thermometers and rulers. There are four that matter here.

The 58% then 52% story is drift. The judge’s bias changed between two weeks. The numbers also differ because of precision: a judge that samples its answers at random gives a slightly different pass rate on every run, even with no update.

Feel the four faults

Here is a model whose true pass rate never changes: 55%. We read it with a grader every week for twelve weeks, 300 questions each time. The three sliders set how the grader misbehaves. Start with all three at zero and note how far the dots still wander. That wander is plain sampling luck, and it is the floor you cannot beat. Then add bias, scatter and drift one at a time.

Read the same model for twelve weeks

Each dot is one week’s pass rate. The blue line is the truth, 55%. Drag the sliders and watch the dots leave the line in three different ways.

0 pts
0 pts
0 pts

Illustrative, not from a paper. Each week reads 300 questions of a model that truly passes 55%. A week’s reading is 55% plus the bias, plus the drift times the week number, plus random scatter you set, plus the sampling luck of 300 questions (about 2.9 points). Seeded, so it is the same on every visit.

Notice what each fault does to the picture. Bias slides the whole cloud up or down. Scatter fans it out. Drift tilts it. They need different fixes. You can subtract a steady bias if you know it. You can shrink scatter by averaging many readings. You cannot average away drift, because it keeps moving. You can only watch for it, and Chapter 9 shows how.

What the rest of the lesson does

A frontier lab runs thousands of evals. Post-training teams use them to decide which checkpoint ships. Agent teams use them to decide whether a change helped. In most of these evals, a model or a rubric does the grading, because people are slow and expensive. So the grader sits between every decision and the truth.

The right question is not “is this judge good?” The right question is “how wrong is this judge, in which direction, and what does that do to the number I am about to act on?” We answer it in layers. First what an LLM judge is and what it gets from a prompt. Then its known biases and how to measure each one. Then how to count agreement with people, what limits that agreement, and how a judge’s mistakes bend a win rate and how to bend it back. Then how to spend a cheap judge and an expensive one wisely, what changes when the grader is a reward model, and last, how to test the grader itself before you ship it.

One rule runs through everything. Treat a judge’s score as a reading, never as the truth. Each chapter gives you a way to put a number on how far the reading may be from the truth.

  1. What an LLM judge is: a prompt in, a verdict out, and the record you keep.
  2. Position bias: why the answer shown first wins, and the swap that exposes it.
  3. Length and self-preference bias: why long answers win, and length control.
  4. Agreement statistics: raw agreement, Cohen’s kappa and Krippendorff’s alpha, by hand.
  5. When people disagree: the ceiling on any grader.
  6. Grader mistakes and the win rate: false positives, false negatives and a worked correction.
  7. Calibration and escalation: a cheap judge, its confidence and an expensive judge.
  8. Reward models as graders: Goodhart’s law and over-optimization.
  9. A meta-eval: a gold set for the grader itself, and criteria drift.
  10. Field guide: the numbers, the checklist and the code.
A judge reads the same model every week and always reports 6 points above the truth, with almost no scatter. Which fault is this, and what can you do about it?

Chapter 1

What an LLM Judge Is

Follow one question and two answers through a judge, and see exactly which record you must keep

A user asks a model: “How do I reverse a list in Python?” Model A answers in two lines with one example. Model B answers in two paragraphs with three methods and a warning about copies. Which answer is better?

You could ask a person. A person needs about two minutes to read both and decide. Your eval has 10,000 such pairs. That is over 330 hours of reading, every time you change a prompt. So teams ask a language model to read the pair instead. It takes seconds and costs cents.

That is the whole idea of an LLM judge: you write a prompt that shows a strong model the question, one or two answers and the grading rules, and you read its verdict. Everything else in this lesson is about knowing when to believe that verdict.

Three ways to ask

There are three common formats. They differ in what the judge sees and what it returns.

In pointwise grading the judge sees one answer and gives it a score, say 1 to 10, or pass or fail. It is cheap: one call per answer. But scores drift between prompts and between days, because the judge has no neighbour to compare against.

In pairwise grading the judge sees two answers to the same question and picks the better one, or calls a tie. Comparing is easier than scoring. People do it more consistently too. The price is order: one answer must come first, and Chapter 2 shows that the first slot has an unfair advantage.

In reference-guided, rubric grading the judge also gets a correct answer or a checklist. G-Eval, a framework for scoring text such as summaries, is the standard example. It asks the judge to write out its own evaluation steps with chain-of-thought (thinking in written steps before answering), then fill in a form with a score for each criterion. With GPT-4 as the judge, G-Eval reaches a Spearman correlation of 0.514 with human judgments on summarization (Liu et al., arXiv 2303.16634). A Spearman correlation compares two rankings: 1 means the same order, 0 means no link. The same paper reports a preliminary finding that LLM-based evaluators may be biased toward text that an LLM wrote.

The device below shows the three formats on our reverse-a-list question. Pick one and read the prompt on top and the record that comes back underneath.

Ask the judge three ways

Choose a format. The first box is what the judge reads. The second is what your code gets back and stores.

Illustrative prompts and records, written for this lesson. Call counts assume one call per answer for pointwise and rubric grading and one per pair for pairwise grading; the second call in the pairwise row is the order swap from Chapter 2.

The data flow, step by step

A judge call is a small pipeline. Your code builds a prompt string from a template. It sends the string to the model. The model returns text. Your code parses that text, which means it cuts the verdict out of the reply and turns it into a typed value. Then your code writes one record to a log.

Two details decide whether the judge is usable. First, ask for a structured reply, such as JSON, and always parse it. A free-text “I think A is slightly better” is hard to count. Second, keep the judge’s reasoning field. It costs a few tokens and lets you read why a verdict looks wrong later.

MT-Bench and Chatbot Arena, the two benchmarks from Zheng et al. (arXiv 2306.05685), studied strong judges of open-ended questions. They report that a strong judge such as GPT-4 reaches agreement above 80% with both controlled and crowdsourced human preferences, which the authors describe as matching human-to-human agreement. That is a real result, and it is why people use judges. The same paper also names the problems we study next: position bias, verbosity bias and self-enhancement bias, plus limited reasoning ability.

pythonimport json

JUDGE_MODEL = "judge-large-2026-09-14"     # a dated version, not "latest"
PROMPT_VERSION = "pair-v3"

def judge_pair(question, a, b, order, call_model):
    first, second = (a, b) if order == "ab" else (b, a)
    prompt = PAIR_TEMPLATE.format(q=question, first=first, second=second)
    raw = call_model(JUDGE_MODEL, prompt, temperature=0)
    try:
        out = json.loads(raw)                  # {"winner": "first"|"second"|"tie", "why": "..."}
        winner = {"first": order[0], "second": order[1], "tie": "tie"}[out["winner"]]
    except (ValueError, KeyError):
        return {"status": "parse_error", "raw": raw}
    return {"status": "ok", "winner": winner, "why": out["why"],
            "order": order, "judge_model": JUDGE_MODEL, "prompt_version": PROMPT_VERSION}

Read the last line. It stores the judge’s exact version and the prompt version in every record. That is not tidiness. When the pass rate jumps by six points next month, those two fields tell you whether the model changed or the instrument did. Chapter 9 uses them.

You log only the winner of each judge call. Next month the win rate shifts by 6 points. What can you not tell?

Chapter 2

Position Bias and the Swap Test

Show the judge the same two answers in both orders, and measure how much the slot decides the verdict

Two friends ask you to taste their soup. You taste Ana’s first, then Ben’s. Both soups are the same recipe. Yet you say Ana’s is slightly better, because the first taste set your idea of “soup” and the second one was only “the same again.”

Language models do something like this when they judge two answers. When the slot an answer sits in changes the verdict, that is position bias. The answer is the same. Only its place in the prompt moved.

The effect can be large. Wang et al. (arXiv 2305.17926) show that the quality ranking of responses can be manipulated by changing their order of appearance in the context. With ChatGPT as the evaluator, Vicuna-13B could be made to beat ChatGPT on 66 of 80 tested queries. That is a model beating itself by shuffling a prompt. Zheng et al. (arXiv 2306.05685) name position bias as one of the main weaknesses of judges as well.

The swap test

Measuring the bias is easy. For each pair, ask the judge twice: once with answer A first, once with answer B first. Then sort the outcomes into three bins.

The share of inconsistent pairs is your flip rate. If the flip rate is 0%, order does not matter. If it is 40%, then on 40 of every 100 pairs the verdict says more about the slot than about the answers.

Here is a worked example with illustrative numbers. We judge 100 pairs of answers from model A and model B, each in both orders.

  1. In 38 pairs the judge picks A in both orders. A wins those cleanly.
  2. In 22 pairs it picks B in both orders. B wins those cleanly.
  3. In 40 pairs it picks whoever sits first. These are the inconsistent ones.

If you ran only the order “A first”, A would win the 38 clean pairs and the 40 slot-driven pairs: 78%. If you ran only “B first”, A would win just the 38: 38%. Same judge, same answers, a 40 point swing. Average the two orders and A gets 38 plus half of 40, which is 58%. The swap cancels the slot’s push, because the slot favours A in one order and B in the other.

Now try it. The device draws 1,000 pairs. Model A is the better model (illustrative: it deserves to win 57% of pairs on content). The judge follows content, except that sometimes it ignores the content and picks the first slot. You set how often.

Swap the order

Three bars: A’s win rate with A shown first, with A shown second, and averaged over both orders. The blue line is what the judge would say with no position bias. Drag the bias up.

30%

Illustrative simulation, not from a paper. On each pair the judge follows content with probability 1 minus the bias, where it picks A 57% of the time, and picks the first slot with probability equal to the bias. The 1,000 pairs are drawn live in your browser.

Watch three things. The first-slot bar climbs and the second-slot bar falls as the bias grows, so a single-order eval would crown a winner by accident. The averaged bar stays near the blue line at low bias, but it slowly slides toward 50%. That slide is the second cost of position bias: even after you cancel the direction, a judge that sometimes ignores content has less signal left. Averaging fixes the tilt. It cannot restore the lost signal.

Fixes, in order of cost

The cheapest fix is to swap and average, as above. Wang et al. call a version of it Balanced Position Calibration: aggregate the results across different orders. It doubles the calls.

A second fix is to make the judge write its evidence before it picks. The same paper calls this Multiple Evidence Calibration: the evaluator generates several pieces of evaluation evidence before it rates. Thinking first gives the content more weight than the slot.

A third fix spends human time only where it pays. The authors define a balanced position diversity entropy that measures how hard an example is, and call on a person when the judge flips. In practice you can do the same with the flip rate: send the inconsistent pairs, and only those, to a person.

pythondef swap_test(pairs, judge):
    clean_a = clean_b = flips = 0
    for q, a, b in pairs:
        v1 = judge(q, a, b, order="ab")["winner"]   # "a", "b" or "tie"
        v2 = judge(q, a, b, order="ba")["winner"]
        if v1 == v2 and v1 != "tie":
            clean_a += v1 == "a"; clean_b += v1 == "b"
        else:
            flips += 1                               # slot-driven or tie
    n = len(pairs)
    win_a = (clean_a + 0.5 * flips) / n              # count flips as half a win
    return {"flip_rate": flips / n, "win_rate_a": win_a}

Report the flip rate next to every pairwise win rate. A win rate of 58% with a 5% flip rate is a different claim from 58% with a 40% flip rate. The second one tells the reader that most of the signal came from half a coin toss.

A judge picks the first answer in 40 of 100 pairs regardless of content, and follows content on the other 60. What does the swap test report as the flip rate?

Chapter 3

Length and Self-Preference Bias

Catch a judge that rewards long answers and its own family, and remove the length effect with a simple regression

A student hands in two essays on the same topic. One is a tight page. The other is four pages and repeats itself. A tired marker sees the thick pile and thinks “more effort.” The long essay gets the better grade.

LLM judges make the same mistake. When a judge likes longer answers regardless of quality, that is verbosity bias, also called length bias. It matters most in leaderboards, because a model can climb by writing more. AlpacaEval, a popular automatic leaderboard, is known to favor models with longer outputs, even though it correlates highly with human preferences (Dubois et al., arXiv 2404.04475).

Why you cannot just say “ignore length”

You can add a line to the prompt: “Do not reward longer answers.” It helps a little. But you cannot trust it, because the bias comes from training, not from the prompt. You have to measure it and correct for it in the numbers.

Length and quality also tangle. A better answer is often longer, because it covers more. So you cannot simply discard long answers. You need a way to ask: “if the two answers had the same length, how often would the judge still prefer A?”

The length-controlled fix, by hand

The Length-Controlled AlpacaEval method fits a generalized linear model. That is a regression that predicts the judge’s preference from the length difference and other features. Then it estimates the preference with the length difference set to zero (Dubois et al.). The paper reports that this raises the Spearman correlation with LMSYS Chatbot Arena from 0.94 to 0.98, and makes the metric more robust to changes in model verbosity.

You can see the idea with a hand example using illustrative numbers. We judge 1,000 pairs of answers, model A against model B. We group them by how much longer A is.

  1. In 600 pairs A is much longer. The judge prefers A in 70% of them.
  2. In 300 pairs the lengths are close. The judge prefers A in 45%.
  3. In 100 pairs A is shorter. The judge prefers A in 30%.

The raw win rate is the weighted average: 0.6 × 0.70 + 0.3 × 0.45 + 0.1 × 0.30 = 0.42 + 0.135 + 0.03 = 58.5%. A looks better than B. But the 600 long pairs drove it. Look only at pairs where the lengths match, and A wins 45%. At equal length, A is the worse model. The regression does this at scale. It uses every pair, but asks what the win rate would be at zero length difference.

The device generates 800 pairs in which model A is truly a little worse (illustrative: its answers are rated lower on content on average), but it writes longer answers. A judge with verbosity bias then prefers A more often than it should. You control how long A writes and how strong the judge’s bias is. The page fits the regression live.

Control for length

Three bars: the raw win rate of model A, the length-controlled win rate from a fitted regression, and the rate you would get on content alone (blue line). Drag the judge’s length bias up.

0.8
0.9 SD

Illustrative simulation, not the AlpacaEval code. For each of 800 pairs, content quality difference d is drawn around 0.2 in favour of B and length difference L around the slider value, in standard deviations. The judge prefers A with probability sigmoid(2d + bias × L). The control fits a logistic regression of the verdict on L and reads off the win rate at L = 0.

With the bias at zero, all three bars agree, up to sampling luck. As the bias grows, the raw bar climbs while the content line stays put. The length-controlled bar stays close to the content line. That gap between raw and controlled is a number you can report: “this leaderboard gain is mostly length.”

Self-preference

A third bias is about who wrote the answer. MT-Bench names it self-enhancement bias: a judge may favor answers written by itself or by models like it (Zheng et al.). G-Eval’s authors report a preliminary finding of the same shape, that LLM-based evaluators may be biased toward LLM-generated text (Liu et al.). Both papers flag it as a risk, so treat it as one.

Test for it directly. Take a gold set where people have ranked answers from several model families. Have your judge rank the same answers. If the judge moves its own family up relative to people, you have measured self-preference. The fixes are plain: judge with a model from a different family than the one under test, or use two judges from two families and average them.

pythonimport numpy as np
from sklearn.linear_model import LogisticRegression

def length_controlled_winrate(a_wins, len_a, len_b):
    """a_wins: 1 if the judge preferred A. Returns the win rate at equal length."""
    d = (np.asarray(len_a) - np.asarray(len_b)).astype(float)
    X = (d / d.std()).reshape(-1, 1)             # length difference, in SDs
    fit = LogisticRegression(C=1e6).fit(X, a_wins)  # nearly unregularised
    b0 = fit.intercept_[0]                       # log-odds when the difference is 0
    return 1 / (1 + np.exp(-b0))
A raw win rate is 58.5%, but among pairs of similar length the same model wins 45%. What is the best reading?

Chapter 4

Agreement Statistics

Count how often a judge and a person agree, then subtract the agreement that luck alone would give

A person and a judge each grade the same ten answers as pass or fail. They agree on seven. “70% agreement” sounds decent. But a judge that says “pass” every time would also agree with a lenient person most of the time. So how good is 70%?

That is the question agreement statistics answer. We go from the simple count to two better measures, using one small example all the way through.

Step 1: raw agreement

Raw agreement (also called percent agreement or observed agreement) is the share of items where both graders gave the same label. Our ten items form a small table, a confusion table, which counts every combination of the two labels.

  1. Both say pass: 5 items.
  2. Person passes, judge fails: 1 item.
  3. Person fails, judge passes: 2 items.
  4. Both say fail: 2 items.

They agree on 5 + 2 = 7 of 10 items, so raw agreement is 0.70. The weakness is plain: it counts agreement that needs no skill.

Step 2: Cohen’s kappa removes luck

Suppose each grader ignored the answers and just said “pass” at its own usual rate. The person passes 6 of 10 (0.60). The judge passes 7 of 10 (0.70). By luck alone they would both say pass with probability 0.60 × 0.70 = 0.42, and both say fail with probability 0.40 × 0.30 = 0.12. Their chance agreement is 0.42 + 0.12 = 0.54. So 54% of the 70% was free.

Cohen’s kappa asks how far the graders are above that free part, as a share of the room that was left:

κ
Kappa. 1 means perfect agreement, 0 means no better than chance, below 0 means worse than chance.
po
Observed agreement: the raw agreement we just counted, 0.70.
pe
Expected agreement by luck, from each grader’s own pass rate: 0.54.

κ = (0.70 − 0.54) / (1 − 0.54) = 0.16 / 0.46 = 0.348

The 70% agreement shrinks to a kappa of 0.35. That is a fair-to-weak signal, not a strong one. No single cutoff is right for every use. A judge that will pick which model ships needs more than a judge that triages a log.

Step 3: Krippendorff’s alpha

Kappa has a limit. It is built for exactly two graders and complete data. Real evals have three people on some items, one on others, and sometimes a 1 to 5 scale. Krippendorff’s alpha handles all of this with one idea: compare the disagreement you saw to the disagreement you would expect if the labels were shuffled across items.

For two graders and a yes or no label, with n labels in total, D items on which they disagree, and np and nf total pass and fail labels, alpha is:

α = 1 − (n − 1) × D / (np × nf)

In our example n = 20 labels, D = 3 disagreeing items, np = 6 + 7 = 13 and nf = 4 + 3 = 7. So α = 1 − 19 × 3 / (13 × 7) = 1 − 57 / 91 = 0.374. It lands close to kappa. It differs slightly because alpha corrects for small samples. With 1,000 items the two would be nearly equal.

Now try the numbers yourself. Change the four counts and watch raw agreement, chance agreement, kappa and alpha move. Then press the second preset, where raw agreement is 90%.

Agreement above luck

The square shows how many items fall in each cell. Rows are the person, columns are the judge. Move the four sliders, or press a preset.

5
1
2
2

Illustrative counts. The first preset is the worked example in the text: raw 0.70, chance 0.54, kappa 0.348, alpha 0.374. Alpha uses the two-grader, yes or no formula above, which matches the general definition for this case.

The second preset shows the trap. Ninety of a hundred items are passes for both graders, five are split each way, and none are failed by both. Raw agreement is 90%. But both graders pass 95% of the time, so luck alone gives 90.5%. Kappa is slightly negative. The 90% number told you almost nothing, because nearly everything was a pass. This is why you must report kappa or alpha beside raw agreement whenever one label dominates.

pythondef agreement(both_pass, p_only, j_only, both_fail):
    """p_only: person passed, judge failed. j_only: judge passed, person failed."""
    n = both_pass + p_only + j_only + both_fail
    po = (both_pass + both_fail) / n
    pp = (both_pass + p_only) / n                    # person pass rate
    jp = (both_pass + j_only) / n                    # judge pass rate
    pe = pp * jp + (1 - pp) * (1 - jp)
    kappa = (po - pe) / (1 - pe) if pe < 1 else float("nan")
    n_labels, d = 2 * n, p_only + j_only
    n_p, n_f = 2 * both_pass + d, 2 * both_fail + d
    alpha = 1 - (n_labels - 1) * d / (n_p * n_f) if n_p and n_f else float("nan")
    return {"raw": po, "chance": pe, "kappa": kappa, "alpha": alpha}

print(agreement(5, 1, 2, 2))   # raw 0.70, chance 0.54, kappa 0.348, alpha 0.374
A judge and a person agree on 90% of items, but both say “pass” on 95% of them. What should you conclude?

Chapter 5

When People Disagree: The Ceiling

Find the best agreement any judge can reach when the humans it is compared with disagree among themselves

Two senior engineers read the same two answers to “explain recursion to a ten-year-old.” One prefers the answer with the story about nesting dolls. The other prefers the short, exact one. Neither is wrong. They have different taste.

Now you ask a judge to agree with them. Whose label should it match? If the two people disagree, the judge cannot agree with both. Some of the gap you see between a judge and “the human” is not the judge’s fault. It is the question’s.

Two kinds of ground truth

Ground truth is the label you treat as correct when you score a grader. It comes in two kinds, and they behave very differently.

Some questions have an objective truth. Does the code pass the tests? Is 17 × 23 equal to 391? Does the summary name the right company? A careful person, or a program, can settle these, so the ceiling for agreement is 100%.

Other questions have a subjective truth, a preference. Which reply is friendlier? Which essay reads better? Reasonable people split. For these, the truth is a distribution of opinions, and two people drawn at random will sometimes disagree.

The difference matters for how we read results. MT-Bench found that a strong judge such as GPT-4 reaches agreement above 80% with human preferences, matching the agreement between humans themselves (Zheng et al., arXiv 2306.05685). That does not say 80% is a weak result. It says people hit about the same level, so a judge has little room to climb past it on preference questions. JudgeBench (Tan et al., arXiv 2410.12784) makes the other point. Existing judge benchmarks mainly measure agreement with human preferences, which can be a poor signal of factual and logical correctness. It builds response pairs labeled by objective correctness in knowledge, reasoning, math and coding, and reports that many strong models, including GPT-4o, do only slightly better than random guessing there.

Put the two together. A judge can match people on taste and still fail on facts. And a judge cannot beat the human ceiling on taste, however good it is.

Compute the ceiling

Here is a simple model, with illustrative numbers. Call an item ambiguous if informed people split about evenly on it, and clear if they all agree. Say 30 of 100 items are ambiguous.

  1. On the 70 clear items, two people agree every time: 70 agreements.
  2. On the 30 ambiguous items, two people agree half the time, as with two coin tosses: 15 agreements.
  3. Human to human agreement is (70 + 15) / 100 = 85%. That is the ceiling.

Now a judge that is right 95% of the time on clear items and tosses a coin on ambiguous ones. Its agreement with a person is 0.70 × 0.95 + 0.30 × 0.50 = 0.665 + 0.15 = 81.5%. It reaches 96% of the ceiling (0.815 / 0.85). If you read “81.5% agreement” as “the judge is wrong 18.5% of the time,” you will chase a gap that mostly cannot be closed.

Use the device to find the ceiling for other mixes. The 10 by 10 grid is your item set. Purple squares are ambiguous, blue are clear.

Find the ceiling

Raise the share of ambiguous items and watch the human ceiling fall. Then raise the judge’s skill on clear items and see how close it can get.

30%
95%

Illustrative model, not from a paper. Clear items: people always agree. Ambiguous items: people split 50 to 50, and the judge also tosses a coin. Real disagreement is graded, not all or nothing, so the true ceiling is a little softer.

What to do about it

First, collect more than one human label per item, at least three on a sample. Store every vote, not only the majority. The share of votes is itself data: a 3 to 0 item is clear, a 2 to 1 item is not.

Second, report the judge on clear items and on ambiguous items separately. On clear items, a miss is a real error. On ambiguous items, a miss is a coin toss, and the right goal is a judge whose confidence matches the human split, not one that always picks a side.

Third, if the labels are objective, make them objective. Run the tests, check the number against a table, or use a reference answer. Use a model judge only for what a program cannot decide.

pythonfrom collections import Counter

def gold_from_votes(votes):
    """votes: list of labels from different annotators for one item."""
    top, k = Counter(votes).most_common(1)[0]
    return {"label": top, "agree": k / len(votes), "clear": k == len(votes)}

items = [gold_from_votes(v) for v in all_votes]
clear = [it for it in items if it["clear"]]
print(f"{len(clear) / len(items):.0%} of items are clear")   # report the judge on each part
Humans agree with each other on 85% of items. A judge agrees with a person on 82%. What is the best reading?

Chapter 6

Grader Mistakes and the Win Rate

Turn a judge’s false positives and false negatives into a correction for the pass rate you report

A smoke alarm that goes off when you make toast is annoying. A smoke alarm that stays silent during a fire is dangerous. Both are mistakes, but they are different mistakes, and you count them separately.

A grader makes the same two kinds of mistake. A false positive is a pass for an answer that is really bad. A false negative is a fail for an answer that is really good. Neither is rare. A lenient judge makes many false positives. A harsh one makes many false negatives.

Two rates describe a grader

To measure them you need a gold set, answers with trusted labels, which Chapter 9 builds. Then count.

Sensitivity (also called recall or the true positive rate) is the share of truly good answers that the judge passes. Specificity is the share of truly bad answers that the judge fails. A judge with sensitivity 0.90 passes 90 of every 100 good answers and wrongly fails 10. A judge with specificity 0.80 fails 80 of every 100 bad answers and wrongly passes 20.

The two rates are independent. A judge can be strict on one side and loose on the other, which is why one accuracy number hides so much.

How errors bend a pass rate

Suppose a model truly passes a share p of tasks. Now watch what the judge reports. Of the p truly good answers, it passes a fraction sens. Of the 1 − p bad answers, it passes a fraction 1 − spec by mistake. The reported pass rate q is the sum:

q
The pass rate the judge reports.
p
The true pass rate, which you want and cannot see.
sens
Sensitivity: the share of truly good answers the judge passes.
spec
Specificity: the share of truly bad answers the judge fails.

With p = 0.60, sens = 0.90 and spec = 0.80: q = 0.60 × 0.90 + 0.40 × 0.20 = 0.54 + 0.08 = 0.62. The judge reports 62% for a model that truly passes 60%. A two point error, in the lenient direction.

Run the equation backwards

You observe q. You know sens and spec from the gold set. Solve for p. Start from the equation, expand, and collect the p terms:

q = p · sens + (1 − p)(1 − spec)  →  q − (1 − spec) = p (sens + spec − 1)  →  p = (q + spec − 1) / (sens + spec − 1)

This is a correction long used in medical testing, where it is often called the Rogan-Gladen estimator. Check it on our numbers: p = (0.62 + 0.80 − 1) / (0.90 + 0.80 − 1) = 0.42 / 0.70 = 0.60. It recovers the truth.

Two warnings come with it. First, if sens + spec is 1, the judge is no better than a coin and the formula divides by zero. The closer the sum is to 1, the weaker the judge, and the more any small error in your sens and spec numbers blows up. Second, the correction removes bias but adds noise. The uncertainty in p is the uncertainty in q divided by (sens + spec − 1). In our example that is a factor of 1 / 0.70 = 1.43. The corrected number is honest, and it has wider error bars (see Eval Statistics and Miller, arXiv 2411.00640, on treating questions as a sample from a larger population).

When the two models are judged unequally

The nastiest case is when the judge makes different mistakes on different models. Say model B writes in a style the judge likes, so its specificity is lower: the judge passes more of B’s bad answers. A gap of a few points in the judge’s favour for B can erase or reverse the true gap. The device lets you switch this on.

Correct the win rate

Blue is the truth, warm is what the judge reports, the green tick is the corrected value. Move sensitivity and specificity. Then switch on the style bias.

0.90
0.80

Illustrative, not from a paper. Model A truly passes 60% and model B 55%. With the style switch on, the judge’s specificity on B is 20 points lower than the slider. The corrected value uses the sensitivity and specificity for each model separately. Values are expected values with no sampling noise; with 500 questions each, the 95% interval on a rate is about 4.3 points before correction.

With the switch off, both models read a little high, the gap shrinks, and the ranking holds. With the switch on and a lenient judge, B can read above A. The correction brings both back to 60% and 55%, but only because we told it the true sens and spec for each model. That is the catch: the correction is only as good as the gold set behind those two numbers, and you need enough gold labels on each model.

pythondef corrected_rate(q, sens, spec):
    denom = sens + spec - 1
    if denom <= 0.05:                       # judge is close to a coin: refuse
        raise ValueError("judge too weak to correct")
    return min(1.0, max(0.0, (q + spec - 1) / denom))

def sens_spec(gold, judged):                # parallel lists of 0/1
    tp = sum(1 for g, j in zip(gold, judged) if g and j)
    tn = sum(1 for g, j in zip(gold, judged) if not g and not j)
    return tp / sum(gold), tn / (len(gold) - sum(gold))

print(corrected_rate(0.62, 0.90, 0.80))     # 0.60
A judge has sensitivity 0.90 and specificity 0.80 and reports a 62% pass rate. What is the corrected pass rate?

Chapter 7

Calibration and Escalation

Let a cheap judge grade everything, and send only the items it is unsure about to an expensive judge

In a hospital emergency room a triage nurse sees every patient in two minutes. Most go home with advice. The few who look serious go straight to a senior doctor. The senior doctor is slow and costly, so the nurse protects that time.

You can run an eval the same way. A cheap judge grades all 10,000 answers. An expensive judge, a bigger model or a person, grades only the answers where the cheap one is unsure. This pattern is called escalation, or a cascade. It only works if the cheap judge knows when it is unsure.

Confidence, and whether to believe it

A judge’s confidence is a number from 0.5 to 1 that says how sure it is of its verdict. You can get it three ways. Ask the judge to state it. Read the probability the model gives to its answer token. Or ask the same question several times, in both orders, and count how often the verdicts agree. The flip rate from Chapter 2 is exactly this last kind of signal.

A confidence is calibrated if it matches reality: of all the verdicts a judge makes at 80% confidence, about 80% should be right. If only 70% are right, the judge is overconfident. Overconfidence is the usual failure.

You measure calibration with a gold set. Sort the judge’s verdicts into confidence bins, then compare each bin’s claimed confidence with its real accuracy. The weighted average gap is the expected calibration error, or ECE. A hand example with illustrative numbers:

  1. Bin 0.6: 20% of verdicts, claimed 0.60, actually right 0.55. Gap 0.05.
  2. Bin 0.8: 50% of verdicts, claimed 0.80, actually right 0.70. Gap 0.10.
  3. Bin 0.95: 30% of verdicts, claimed 0.95, actually right 0.80. Gap 0.15.

ECE = 0.2 × 0.05 + 0.5 × 0.10 + 0.3 × 0.15 = 0.01 + 0.05 + 0.045 = 0.105. The judge claims about 10 points more accuracy than it has. Its most confident bin is the most wrong.

Where calibration matters in a cascade

There are two jobs here, and they need different things.

The first job is choosing which items to escalate. That only needs the confidence to be ranked correctly: lower confidence should mean more likely wrong. Even an overconfident judge can do this well.

The second job is choosing how many. You want to say “escalate until the whole eval is 92% accurate.” To hit a target you must trust the numbers. If the cheap judge claims 95% when it is right 80%, you will escalate too few items and miss your target without knowing it. The fix is to recalibrate on the gold set, which means mapping each raw confidence to the accuracy you measured in that bin.

The device simulates 1,000 items. The cheap judge costs 1 unit per item and always runs. The expensive judge costs 20 units and is right on 95% of what it sees. You set the confidence threshold below which items are escalated, and how overconfident the cheap judge is.

Set the escalation threshold

The solid warm line is the accuracy you really get at each cost. The dashed purple line is what the cheap judge’s confidence claims. The gap between them is overconfidence. Move the threshold, then lower the calibration.

0.85
0.75

Illustrative simulation, not from a paper. Cheap judge confidence runs from 0.5 to 1 across 1,000 items; its true accuracy is 0.5 plus calibration times (confidence minus 0.5), so calibration 1.00 means it is exactly calibrated. Cost counts 1 unit per item for the cheap judge plus 20 per escalated item. Values are expected values, no sampling noise.

Slide the threshold from 0.5 up to 1. At 0.5 nothing escalates: cost is 1 unit per item and accuracy is what the cheap judge gives alone. At 1.0 everything escalates: cost is 21 units and accuracy is the expensive judge’s 95%. Between those, the curve bends. A small amount of escalation buys a lot of accuracy, because the least-confident items are the ones the cheap judge gets wrong. That bend is the whole value of a cascade.

Now lower the calibration. The solid line barely moves, because ranking still works. The dashed line shifts above it. The cheap judge now claims accuracy it does not have, and a threshold chosen from its own claims lands short of the goal.

What an expensive judge cannot do

Escalation has a hard limit. The expensive judge is also a judge. JudgeBench builds hard response pairs labeled by objective correctness and finds that many strong models, including GPT-4o, perform only slightly better than random guessing (Tan et al., arXiv 2410.12784). On such items the second opinion is a second guess. When the labels are checkable, escalate to a check: run the code, compute the sum, query the table. When they are not, escalate to people.

Wang et al. (arXiv 2305.17926) propose the same shape for position bias. Their Human-in-the-Loop Calibration uses a balanced position diversity entropy to measure how hard an example is, and brings in human help when needed. The escalation rule is the part that carries over: spend the scarce resource where the judge’s own signals say it is lost.

pythondef cascade(item, cheap, expensive, tau):
    v = cheap(item)                       # {"verdict": "pass", "conf": 0.91, "flip": False}
    if v["conf"] >= tau and not v["flip"]:
        return {**v, "by": "cheap", "cost": 1}
    w = expensive(item)                   # bigger model, executed check, or a person
    return {**w, "by": "expensive", "cost": 21}   # cheap call already paid for
A cheap judge claims 95% confidence on a bin where it is right 80% of the time. What does this break in a cascade?

Chapter 8

Reward Models and Over-Optimization

See what happens when a model is trained to please a grader, and why the score keeps rising after the quality falls

A teacher tells a class: “I will pay a sweet for every correct spelling on the board.” Within a week one pupil has learned to write only three-letter words. The count of sweets goes up. Spelling does not improve.

That is Goodhart’s law: when a measure becomes a target, it stops being a good measure. Everything so far treated the grader as a thermometer. Here the grader becomes a coach. Someone, or something, is now searching for the answers it scores highest, and the search finds the grader’s blind spots first.

A reward model is a learned grader

In post-training, a reward model (RM) is a model trained on human preference data to give any answer a single score. A language model is then improved to produce answers the RM scores highly. Two common ways to do it are reinforcement learning, which adjusts the model’s weights step by step, and best-of-n sampling, which draws n answers and keeps the highest-scoring one. An LLM judge used as a reward does the same job under a different name.

The RM is a proxy. It was trained to imitate human preference, so it is right on average on ordinary answers. But the optimizer is not looking at ordinary answers. It is hunting for the rare answer with the highest score, and the highest scores are where an imitation is most likely to be wrong.

Gao, Schulman and Hilton (arXiv 2210.10760) measured this cleanly. They used a fixed gold-standard reward model as a stand-in for human labelers, and used it to label the data for a smaller proxy RM. Then they optimized against the proxy and watched the gold score. Their results: optimizing the proxy too hard can hurt the gold score, which they tie to Goodhart’s law; the shape of the gold-versus-optimization curve differs between reinforcement learning and best-of-n; and the fitted coefficients scale smoothly with the number of RM parameters. They also vary the RM’s data size, the policy size and the KL penalty, a term that keeps the policy near its starting point.

A hand example of the turn

Take four candidate answers. The RM gives each a score. We also know the true quality, which is the gold score. The numbers are illustrative.

  1. Answer 1: RM 1.2, gold 0.9.
  2. Answer 2: RM 1.5, gold 1.1.
  3. Answer 3: RM 3.8, gold −0.8. It flatters the reader and sounds confident, but it is wrong.
  4. Answer 4: RM 0.3, gold 0.2.

With best-of-2 on answers 1 and 2, the RM picks answer 2: gold 1.1. With best-of-4, it picks answer 3: gold −0.8. More search made the RM score higher (3.8 against 1.5) and the real quality lower. The RM did nothing different. It simply got offered a bigger menu.

Search harder against the grader

The purple line is the reward model’s score of the chosen answer. The blue line is its true quality. Each step right doubles the number of candidates. Raise the exploit rate and size and watch the lines part.

4.0%
4.0
64

Illustrative toy, not a replication of Gao et al.; their curves come from fitted reward models. Here each candidate has true quality drawn from a standard normal. A share of candidates are exploits: true quality about −1, but the proxy adds the bonus. The proxy also adds noise of 0.5. Each point averages 300 seeded trials.

With the exploit rate at zero, both lines rise together, and the gap between them is just the noise the search selects on. With some exploits and a bonus, the purple line keeps climbing while the blue line peaks and falls. The peak is the best amount of search you can afford. Past it, every extra sample buys score and loses quality.

What helps

None of these ends the problem. Each moves the peak to the right.

pythondef best_of_n(prompt, policy, rm, n):
    cands = [policy(prompt) for _ in range(n)]
    scores = [rm(prompt, c) for c in cands]           # proxy: this is what you optimise
    return cands[max(range(n), key=scores.__getitem__)]

def pick_stop(ckpts, heldout_judge, probe_set):
    """Stop where a grader the optimiser never saw peaks, not where the RM score peaks."""
    curve = [(c, sum(heldout_judge(p, c(p)) for p in probe_set) / len(probe_set)) for c in ckpts]
    return max(curve, key=lambda t: t[1])[0]
During best-of-n, the reward model score of the chosen answer keeps rising as n grows, but a held-out human check shows quality has started to fall. What is happening?

Chapter 9

A Meta-Eval: Testing the Grader Itself

Build a labelled gold set for the grader, size it with an error bar, and re-run it on a schedule to catch drift

A lab keeps a small steel block in a locked drawer. Its mass is known to a hair. Every Monday, a technician weighs it on the lab’s scale. If the scale says anything else, the scale is repaired before any other weighing is trusted.

A meta-eval is that steel block for a grader. It is an eval whose subject is the grader itself. You hold a set of items with trusted labels, run the grader on them, and score the grader. You do this before you ship it, and again on a schedule.

What goes in the gold set

A gold set is the list of items with trusted labels. Each record should carry enough to slice the results later.

json{ "id": "g-0412",
  "question": "Why does my training loss blow up after step 800?",
  "answer": "...",                       // the model output being graded
  "votes": ["fail", "fail", "pass"],     // every annotator, not just the majority
  "label": "fail", "clear": false,       // from Chapter 5
  "slice": {"task": "debugging", "length": "long", "source_model": "family-b"},
  "rubric_version": "r3", "labelled_on": "2026-09-30" }

Three rules keep it honest. Split the gold set in two: a dev half you may use to tune the judge prompt, and a held-out half you touch only to report. If you tune on the half you report, the number is flattering. Cover the slices that matter: long and short answers, each task type, each model family. A judge that is 90% right on average and 60% right on long answers is a bias hiding in a mean. And store the rubric version, because the rules change.

How many labels is enough?

The accuracy you measure on the gold set is itself a sample. Measure 84% on 50 items and the true figure might be 75% or 90%. A confidence interval is a range that holds the true value most of the time (here 95% of the time). For a rate measured on n items, a good small-sample choice is the Wilson interval.

Worked by hand for 84% on n = 50. The 1.96 below is the standard multiplier for a 95% interval, and its square is 3.84.

  1. Denominator: 1 + 3.84 / 50 = 1.0768.
  2. Centre: (0.84 + 3.84 / 100) / 1.0768 = 0.8784 / 1.0768 = 0.816.
  3. Half-width: 1.96 × √(0.84 × 0.16 / 50 + 3.84 / 10,000) / 1.0768 = 0.101.
  4. Interval: 0.816 ± 0.101, so about 71.5% to 91.7%.

That interval is 20 points wide. It cannot tell a 75% judge from a 90% one. With 500 items and the same 84%, the interval is about 80.5% to 87.0%. This is the same error-bar reasoning Miller (arXiv 2411.00640) applies to any eval: questions are a sample from a larger population, so every score deserves an interval. Our lesson on eval statistics goes deeper.

Use the device in two modes. In the first, pick how many gold labels you have and see the interval. In the second, watch a weekly check on a smaller canary set catch, or miss, a silent judge update.

Size the gold set, watch for drift

50
84%

Illustrative. The size view uses the Wilson 95% interval. In the drift view the true judge accuracy is 88%, falls to 80% from week 7 if the update switch is on, and each week the check scores that many canary items. The alarm line is 88% minus 1.96 standard errors for that many items. Draws are random in your browser.

In the size view, the interval narrows with the square root of n. To halve the width you need four times the labels. In the drift view, a small canary set misses a real 8 point fall, because weekly luck is as large as the fall. A larger one catches it the week it happens. That is the trade: labels cost money, and every missed drift costs trust in every number downstream.

Criteria drift: the rubric moves while you label

Shankar et al. (arXiv 2404.12272, “Who Validates the Validators?”) describe a trap they call criteria drift. People need criteria to grade outputs, but grading outputs helps them discover what the criteria should be. They also find that some criteria depend on the specific outputs observed, so you cannot write the rubric first and fix it forever.

Their system, EvalGen, asks users to grade a subset of outputs and uses that feedback to choose the evaluator implementations that best match the grades. The practical lessons for your own gold set:

pythonfrom math import sqrt

def wilson(k, n, z=1.96):
    p = k / n; d = 1 + z * z / n
    centre = (p + z * z / (2 * n)) / d
    half = z * sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
    return centre - half, centre + half

def audit(judge, gold, floor):
    hits = [judge(g) == g["label"] for g in gold if g["clear"]]
    lo, hi = wilson(sum(hits), len(hits))
    return {"acc": sum(hits) / len(hits), "ci": (lo, hi), "ship": lo >= floor,
            "judge_model": judge.version, "rubric": gold[0]["rubric_version"]}
A judge scores 84% on 50 gold items, with a 95% interval of about 72% to 92%. You need to know whether it is at least 85% accurate. What do you do?

Chapter 10

Field Guide: Shipping a Grader

Keep the numbers, the checklist and the audit code in one place, and know where to read next

The one-paragraph summary

A grader is a measuring instrument, so it has accuracy, precision, bias and drift. An LLM judge has three known biases: it favors the slot an answer sits in (position), longer answers (verbosity) and answers like its own (self-preference). You measure the first with a swap test and the second with a length-controlled regression. You count agreement above luck with Cohen’s kappa or Krippendorff’s alpha, and you read it against the ceiling set by how much people disagree. Judge mistakes bend a pass rate in a direction you can compute, and you can correct it from the judge’s sensitivity and specificity, at the price of a wider error bar. A cheap judge plus an expensive one is a good cascade if you recalibrate the confidence. When a grader becomes a reward, the optimizer finds its blind spots, so you watch a held-out grader. And the grader itself needs an eval: a gold set with votes and slices, an interval on its accuracy, and a weekly canary.

The numbers that matter

WhatNumberWhy it matters
Judge to human agreement (MT-Bench)Above 80% for GPT-4, matching human to human (arXiv 2306.05685)Strong judges are usable, and 80% is near the human level
Order alone (unfair evaluators)Vicuna-13B beats ChatGPT on 66 of 80 queries with ChatGPT judging (arXiv 2305.17926)Always swap the order and report the flip rate
G-Eval on summarizationSpearman 0.514 with humans using GPT-4 (arXiv 2303.16634)A form-filling rubric ranks well, but not perfectly
Length control (AlpacaEval)Spearman with Chatbot Arena 0.94 to 0.98 (arXiv 2404.04475)Removing the length effect makes a leaderboard track people better
Objective correctness (JudgeBench)Many strong models, GPT-4o among them, only slightly above random (arXiv 2410.12784)Agreement with taste does not prove a judge can check facts
Worked agreement exampleRaw 0.70, chance 0.54, kappa 0.348, alpha 0.374 (illustrative)Subtract luck before you trust a percentage
Worked correctionq 0.62, sens 0.90, spec 0.80 gives p 0.60 (illustrative)Judge errors move a rate by points, and you can undo it
Gold set size84% on 50 items: 71.5% to 91.7%. On 500: 80.5% to 87.0% (illustrative)Fifty labels cannot separate a 75% judge from a 90% one
Over-optimization (Gao et al.)Proxy score rises while gold score can fall (arXiv 2210.10760)Stop at the peak of a held-out grader
Criteria drift (Shankar et al.)Criteria change as people grade outputs (arXiv 2404.12272)Version the rubric and keep a never-relabeled canary

The ship checklist

Tick what you have done for the grader you plan to ship. The read-out names the first gaps.

Is this grader ready?

The checklist is this lesson’s own summary, one item per chapter, and is not taken from a paper. Ticks stay on this page only.

The core as code

pythondef grader_audit(judge, gold, floor=0.85):
    """One report that covers Chapters 2, 4, 6 and 9."""
    rows = []
    for g in gold:                                        # g["a"], g["b"], g["label"] in {"a","b"}
        v1 = judge(g, order="ab")["winner"]
        v2 = judge(g, order="ba")["winner"]
        rows.append((g, v1, v2))
    flip = sum(v1 != v2 for _, v1, v2 in rows) / len(rows)            # position bias (Ch 2)
    clear = [(g, v1) for g, v1, v2 in rows if v1 == v2 and g["clear"]]
    hits = sum(v == g["label"] for g, v in clear)
    lo, hi = wilson(hits, len(clear))                                 # error bar (Ch 9)
    po = hits / len(clear)
    pj = sum(v == "a" for _, v in clear) / len(clear)                 # judge picks a
    pg = sum(g["label"] == "a" for g, _ in clear) / len(clear)        # gold picks a
    pe = pj * pg + (1 - pj) * (1 - pg)
    kappa = (po - pe) / (1 - pe)                                      # agreement above luck (Ch 4)
    return {"flip_rate": flip, "acc": po, "ci": (lo, hi), "kappa": kappa,
            "ship": lo >= floor and flip < 0.15 and kappa > 0.5}

Where this sits, and what to read next

Grader reliability is the part of evaluation that asks whether the instrument can be trusted. The rest of evaluation asks what to measure and how to read the number.

Sources

Every number labeled as a paper’s was read on the abstract page of that paper. Toy numbers in the devices are labeled illustrative.

The takeaway. A score is a reading, not a fact. Before you act on it, know the grader’s bias, its error rates, how much people disagree, and how wide the error bar is. Then test the grader the way you test the model: on labels it has not seen.
You must report one win rate for a new model, graded by an LLM judge, to a team that will act on a 3 point difference. Which package is most defensible?

Now press Present or Teach and explain, from memory, how you would decide whether to trust one judge’s score. Then go back to the top and run the four graders again: this time say which fault each one has.

Grader Reliability: LLM Judges You Can Trust
MT-Bench · Length-Controlled AlpacaEval · Reward model overoptimization · Who Validates the Validators · Back to Gleams