A 99%-accurate fraud detector that never catches fraud is not a paradox — it is a metric you designed badly. This lesson teaches you to build numbers that cannot lie to you: confusion matrices, ROC and PR curves, calibration, ranking scores, honest aggregation, and the failure mode where a good metric goes bad the moment you optimize it.
A payments company processes 10,000 transactions a day. Roughly 100 of them are fraudulent — about 1%. It is quarterly-review day, and a junior engineer flips to a slide: the new fraud model scored 99% accuracy. The room nods. Someone says "ship it." Nobody in that meeting has noticed that the model has never, not once, caught a fraudulent transaction.
How is that possible? The engineer's "model" is one line long: predict legitimate for every transaction. It never flags anything. And because 9,900 of the 10,000 transactions really are legitimate, it is correct 9,900 times out of 10,000. That is accuracy — the fraction of predictions that are correct — and here it equals 9,900 / 10,000 = 0.99.
Read that again slowly, because the trap is subtle. Accuracy did not reward the model for understanding fraud. It rewarded the model for agreeing with the majority. When one class makes up 99% of your data, agreeing with the majority is worth 99 points before the model has understood a single thing about the problem. Accuracy silently inherits the class distribution.
This villain has a name: class imbalance — when the classes you are predicting appear at wildly different rates. And it is not a rare edge case. Fraud, disease, manufacturing defects, security intrusions, the relevant documents in a search index — almost every problem worth solving is imbalanced, because the interesting events are, by their nature, rare. If the interesting thing happened half the time, you would not need a model to find it.
Here is the mental move that runs through this entire lesson. When you are handed a number, ask: what decision does this number drive, and what does each kind of mistake cost? A missed fraud costs the company a chargeback — real money out the door. A false alarm costs a support call and an annoyed customer. Accuracy prices both of those at zero when the positive class is rare, because it can hit 99% while getting every consequential case wrong.
Now watch the metric do something genuinely perverse. A second engineer builds a model that actually tries. It catches 10 of the 100 frauds — not great, but real signal — while raising 50 false alarms. Let us score it. It gets 10 frauds right, misses 90, correctly passes 9,850 legitimate transactions, and wrongly flags 50. That is (10 + 9,850) / 10,000 = 98.6% accuracy. Lower than the do-nothing model. By the accuracy leaderboard, trying to catch fraud is a regression.
That is the moment to feel the metric actively lying to you. A number that ranks "catch zero frauds" above "catch ten frauds" is not measuring the thing you care about. It is answering a question you did not mean to ask.
Step back to the general principle. A metric is a lossy compression of everything a model does into a single number. Every compression throws information away — that is what "lossy" means. Metric design is choosing what you can afford to lose. Accuracy chose to throw away the distinction between the two error types, and on an imbalanced problem that was exactly the wrong thing to discard.
The rest of the lesson is a toolkit for making that choice deliberately: per-class counting so no error can hide (Ch1), threshold-free views of a scoring model (Ch2), scoring the probabilities themselves (Ch3), continuous-target metrics (Ch4), ranking and retrieval (Ch5), honest aggregation across classes and slices (Ch6), and the failure mode where a good metric decays because you optimized it (Ch7). Then a playground (Ch8) where every metric reacts at once, and an interview room (Ch9).
One boundary, stated up front so we never blur it. This lesson designs the metric definitions. Whether a measured difference is real — whether 84.3% genuinely beats 83.9% or is just noise, how big a confidence interval is, how many samples you need — is the job of the sibling lesson, The Statistics of Evaluation, which you should have met already. Design lives here; significance lives there. Every honest claim needs both halves.
A field of transactions: teal = legitimate, red = fraud. Two dials below: accuracy and recall (frauds caught). Toggle between the lazy model (flags nothing) and the trying model (flags some), then drag the fraud rate from a balanced 50% down to a realistic 0.5%. Watch the lazy model's accuracy dial climb toward 99+ while its recall stays pinned at zero — and watch the trying model score lower accuracy even as its recall shows real signal.
Worked example — the paradox in exact numbers. Take 10,000 transactions, 100 fraudulent (1%). Model A predicts LEGITIMATE always. Model B flags 60 transactions: 10 are true frauds, 50 are false alarms. Let us grind every number by hand.
Model A is correct on all 9,900 legitimate transactions and wrong on all 100 frauds:
But recall — the fraction of actual frauds the model caught — tells the real story:
Now Model B. First we count the four outcomes. Of the 60 flags, 10 are correct (true positives, TP) and 50 are wrong (false positives, FP). Of the 100 frauds, 10 were caught, so 90 were missed (false negatives, FN). Of the 9,900 legitimate transactions, 50 were wrongly flagged, leaving 9,850 correctly passed (true negatives, TN):
Model B's accuracy adds the two "correct" cells:
Its recall — coverage of the real frauds:
And its precision — the fraction of its flags that were real:
The verdict is stark. Model B has lower accuracy (0.986 < 0.99) yet catches 10 frauds instead of 0. The accuracy leaderboard ranks the useless model first. Every downstream metric in this lesson exists to stop that from happening.
Let us reproduce the paradox in code, first by literally counting — the same arithmetic your hand just did — then with a library so you can trust it in production.
python # From scratch: count, don't trust a library yet def accuracy_and_recall(y, pred): correct = caught = missed = 0 for yi, pi in zip(y, pred): correct += (pi == yi) # any correct prediction caught += (pi == 1 and yi == 1) # TP missed += (pi == 0 and yi == 1) # FN acc = correct / len(y) rec = caught / (caught + missed) if (caught + missed) else 0.0 return acc, rec # 10,000 labels: 100 frauds (1), 9,900 legit (0) y = [1]*100 + [0]*9900 lazy = [0]*10000 # flags nothing trying = [1]*10 + [0]*90 + [1]*50 + [0]*9850 print(accuracy_and_recall(y, lazy)) # (0.99, 0.0) print(accuracy_and_recall(y, trying)) # (0.986, 0.10)
python # Library equivalent — same answers from sklearn.metrics import accuracy_score, recall_score print(accuracy_score(y, lazy), recall_score(y, lazy)) # 0.99 0.0 print(accuracy_score(y, trying), recall_score(y, trying)) # 0.986 0.1
Accuracy compressed four numbers into one and lied. So in the next chapter we refuse to compress — yet — and look at all four numbers directly.
Accuracy hid two error types inside one number. The cure is to stop hiding them. Every binary prediction lands in one of exactly four buckets, arranged in a grid of what the model said versus what was true. That grid is the confusion matrix, and every binary metric you will ever compute is just some ratio of its four cells.
Here is the naming convention, because it trips everyone up once. The second letter is what the model said (Positive = flagged, Negative = passed). The first letter is whether it was right (True = correct, False = wrong). So TP (true positive) = flagged and it really was fraud; FP (false positive) = flagged but it was legitimate, a false alarm; FN (false negative) = passed but it was fraud, a miss; TN (true negative) = passed and it really was legitimate.
The two error cells — FP and FN — are two different victims, and this is where metric design actually begins. A false positive hurts the innocent user whose card gets frozen at the grocery checkout. A false negative hurts the company, which eats the chargeback on the fraud it let through. Before choosing any metric, ask: which victim does my product fear more?
Two metrics fall straight out of the grid, and they answer two genuinely different questions. Precision = TP / (TP + FP): "of everything I flagged, how much was real?" — the trustworthiness of an alarm. Recall = TP / (TP + FN): "of everything real, how much did I catch?" — coverage. Same numerator, different denominator, different question.
These two are in tension by construction. Flag everything and recall shoots to 1.0 — you caught every fraud — but precision collapses toward the base rate, because almost everything you flagged was legitimate. Flag only the single most certain case and precision hits 1.0 while recall collapses to near zero. Any single-number summary of a classifier has to referee this trade-off somehow.
The most common referee is the F1 score, and its exact form is not arbitrary — it is a harmonic mean of precision and recall for a specific reason. Suppose precision is 0.9 and recall is 0.1. The ordinary arithmetic mean is (0.9 + 0.1) / 2 = 0.5 — a passing grade for a model that misses 90% of the fraud. The harmonic mean refuses to be so generous.
The harmonic mean stays close to the worse of the two numbers. You cannot buy back a dead recall with a shiny precision; a 0.9 precision cannot subsidize a 0.1 recall. That is the whole point of choosing harmonic over arithmetic: it blocks the subsidy.
Why does the harmonic mean behave that way? The intuition lives in rates. Drive one leg of a round trip at 90 mph and the return leg at 10 mph, and your average speed is nowhere near 50 — it is dominated by the slow leg, because you spend so much longer on it. F1 treats precision and recall as two legs of one journey, so the slow leg dominates. A model is only as good as its weakest half.
Sometimes equal weighting is wrong, and the F-beta score is the tunable referee. Beta > 1 weights recall more heavily (use F2 for medical screening, where a miss is catastrophic); beta < 1 weights precision more (use F0.5 for spam filters, where a false alarm burns user trust). F-beta collapses toward whichever error type you told it to fear.
And here is the design question F1 quietly answers for you: F1 asserts that a false positive and a false negative cost the same. If your product disagrees — if a missed fraud costs 4x an annoying support call — F1 is the wrong referee. Pick beta from the actual cost ratio, or keep precision and recall separate on the dashboard and refuse to blend them at all.
A population of scored items streams through a threshold gate into four proportional tiles: TP, FP, FN, and dim TN. Drag the threshold to watch items re-sort and the metrics move. Drag beta to morph the F-beta bar in real time, or hit a preset scenario to see where its cost structure wants the threshold. Notice the callout when you push into the extreme-precision corner: arithmetic mean stays cheerful at ~0.5 while F1 sinks toward 0.18.
Worked example — a content-moderation classifier. On 1,000 posts: TP = 40, FP = 10, FN = 60, TN = 890. Step through every metric.
Accuracy looks fine at 0.93, but it is quietly absorbing the 60 missed posts. Now F1:
Now the famous extreme case, P = 0.9 and R = 0.1, to see the harmonic mean earn its keep. Arithmetic mean:
The harmonic mean sits near the worse number. Now push beta to make it harsher on the failing dimension. With beta = 2 (recall-weighted, so β2 = 4):
Even harsher, because beta = 2 cares most about recall, which is the dead dimension. And with beta = 0.5 (precision-weighted, β2 = 0.25):
More forgiving, because beta = 0.5 rewards the strong dimension (precision). The referee bends toward whatever you told it to value.
python # From scratch: build the matrix, then metrics — print the intermediates def confusion(y, pred): TP = FP = FN = TN = 0 for yi, pi in zip(y, pred): if pi == 1 and yi == 1: TP += 1 elif pi == 1 and yi == 0: FP += 1 elif pi == 0 and yi == 1: FN += 1 else: TN += 1 return TP, FP, FN, TN def fbeta(P, R, beta): b2 = beta*beta num = (1 + b2) * P * R den = b2 * P + R print(" num", round(num,4), "den", round(den,4)) # mirror the hand math return num / den P, R = 0.9, 0.1 print(fbeta(P, R, 1), fbeta(P, R, 2), fbeta(P, R, 0.5)) # 0.18 ... 0.1216... 0.3462...
python # Library equivalent from sklearn.metrics import confusion_matrix, fbeta_score # confusion_matrix returns [[TN, FP], [FN, TP]] print(fbeta_score(y, pred, beta=2)) # recall-weighted F2
Everything here depended on where we put the threshold. But a model does not output decisions — it outputs scores, and the threshold is a choice we make afterwards. Next we evaluate the scores themselves, before any threshold.
Reframe what a classifier really is. Under the hood it does not hand you a decision — it hands you a score for each example (a fraud probability, a relevance weight). A threshold turns scores into decisions, and that threshold is a business choice applied afterwards. So a scoring model is not one classifier; it is a whole family of classifiers, one for every possible threshold.
This is why comparing two models at a single arbitrary threshold is treacherous: you are conflating model quality (how well it ranks) with threshold choice (a knob you can turn later). We want to evaluate the family all at once.
The ROC curve does exactly that, and it is built mechanically. Sort your examples by score, highest first. Now sweep the threshold downward, admitting one example at a time. After each step, plot a point: the x-axis is the false positive rate (FPR = false alarms so far / all negatives) and the y-axis is the true positive rate (TPR = catches so far / all positives, which is just recall). Each positive you admit steps the curve up; each negative steps it right. The curve is literally a picture of your sorted ordering.
The area under that curve is the AUC (Area Under the ROC Curve), and it has a beautiful, exact meaning that most people never learn: AUC is the probability that a randomly chosen positive scores higher than a randomly chosen negative. It is a measure of ranking quality, and we can prove it by counting.
Take 5 positives and 5 negatives. There are 5 × 5 = 25 positive-negative pairs. Count the pairs where the positive outscores the negative. If 21 of the 25 pairs are "wins," then AUC = 21 / 25 = 0.84 — and that number equals the geometric area under the ROC curve, exactly. This pair-counting identity is the single most useful fact about AUC.
Read the corners of ROC space to build intuition. The point (0, 0) is "flag nothing." The point (1, 1) is "flag everything." The diagonal line from corner to corner is a coin flip: AUC = 0.5, no ranking ability at all. And AUC below 0.5 is not a broken model — it is an informative model with its sign flipped; negate the scores and it works.
Now the crucial companion. The PR curve is the same threshold sweep, but plotted in (recall, precision) space instead of (FPR, TPR) space. The structural difference is everything: ROC's x-axis (FPR) divides false alarms by the count of negatives. On a 1000:1 imbalanced problem, the negative count is enormous, so 500 false alarms barely nudge FPR — while those same 500 false alarms crater precision, because precision divides by the (tiny) count of things you flagged.
So the rule of thumb writes itself. When positives are rare and you care about the quality of your flags — fraud, retrieval, moderation — the PR curve, and its area (average precision), is the honest picture; ROC can look near-perfect on the same model. When classes are balanced, or you care about both error types symmetrically, ROC is fine and carries the cleaner probabilistic meaning.
One design warning about AUC. It integrates over all thresholds, including absurd ones the product will never use. If your product only ever operates at high precision, a partial AUC (over the region you actually live in) or a precision-at-fixed-recall target measures the part of the curve that pays your rent.
Left: 10 examples on a score axis (teal = positive, red = negative) with a draggable threshold gate. Right: the ROC curve, with the current threshold's point pulsing. Hit Show all 25 pairs to watch the pair-counting proof animate to 21/25 = 0.84. Flip to the PR view, then toggle ×50 negatives to see ROC barely twitch while the PR curve collapses — the imbalance lesson made visible.
Worked example — 10 scored examples. Positives (5): scores 0.9, 0.8, 0.7, 0.5, 0.3. Negatives (5): scores 0.6, 0.4, 0.35, 0.2, 0.1. Let us count the winning pairs to get AUC by hand.
For each positive, count how many of the 5 negatives it outscores:
Now two operating points on the same model, to see how much the threshold changes the P/R story. At threshold 0.65 we flag {0.9, 0.8, 0.7} — all three positive:
Lower the threshold to 0.45 and we flag {0.9, 0.8, 0.7, 0.6, 0.5}. Four are positive (0.9, 0.8, 0.7, 0.5); one is a negative (0.6):
Same model, two thresholds, two very different (P, R) stories — and the curve holds them all at once.
python # From scratch: AUC two ways, assert they agree import numpy as np s = np.array([0.9,0.8,0.7,0.5,0.3,0.6,0.4,0.35,0.2,0.1]) y = np.array([1,1,1,1,1,0,0,0,0,0]) # Method 1: count positive-beats-negative pairs pos, neg = s[y==1], s[y==0] wins = sum(1 for p in pos for n in neg if p > n) print(wins, wins / (len(pos)*len(neg))) # 21 0.84
python # Library — same 0.84 from sklearn.metrics import roc_auc_score, precision_recall_curve, average_precision_score print(roc_auc_score(y, s)) # 0.84 print(average_precision_score(y, s)) # PR-curve area, the honest one when rare
ROC and PR evaluated the ordering of scores. But a triage system does not just rank patients — it tells the nurse "80% risk." Is that number a measurement or a vibe? That is calibration, and it is next.
A hospital runs a sepsis-risk model. For an incoming patient it outputs 0.8, and the hospital protocol keys real decisions off that number — drug dosing, bed allocation, how fast a nurse is paged. So ask the uncomfortable question: among all the patients this model scores 0.8, do 80% of them actually deteriorate? If only half do, then "0.8" is not a probability. It is a mislabeled ranking score, and every downstream expected-cost calculation built on it is wrong.
This is calibration, and its definition is precise: among all predictions with confidence p, the fraction that turn out positive should be p. Confidence 0.8 events should happen 80% of the time; confidence 0.3 events, 30% of the time.
The stunning fact is that calibration is completely orthogonal to ranking. You can have perfect AUC and terrible calibration — imagine all your scores squished near 0.5 but in exactly the right order; the ranking is flawless, the probabilities are garbage. And you can have decent calibration with mediocre AUC. Ranking is "did you get the order right?"; calibration is "did you get the odds right?" — two different skills.
The tool that exposes calibration is the reliability diagram, built from scratch as follows. Bin the predictions by their confidence. For each bin, plot the average confidence (x) against the observed positive rate (y). Perfect calibration is the diagonal line y = x. Points sagging below the diagonal mean the model is overconfident (it said 0.8, reality delivered 0.5); points bowing above mean underconfident.
To turn the diagram into a single number, compute the Expected Calibration Error (ECE) — a weighted average of the per-bin gaps between confidence and observed frequency. It is nothing more mysterious than that: sum over bins of (bin's share of the data) × |average confidence − observed rate|.
ECE has design fragilities you must know as its author. The bin count is a free parameter that changes the answer — 3 bins and 15 bins give different ECE on identical data. And rare-confidence bins are noisy, with few points to average. This is exactly why careful papers report ECE with the binning scheme attached; an ECE quoted without its bin count is under-specified.
A bin-free alternative is the Brier score = the mean squared error between the predicted probability and the 0/1 outcome. It is a proper scoring rule, and "proper" means something important: a model minimizes its expected Brier by reporting its true belief. It cannot be gamed by hedging or by sounding confident — unlike accuracy, which rewards a confident guess exactly as much as confident knowledge.
The Brier score decomposes (the Murphy decomposition): Brier = calibration (reliability) − discrimination (resolution) + irreducible outcome uncertainty. Two models can share a Brier score while one is a sharp liar (great resolution, poor reliability) and the other a vague truth-teller (poor resolution, great reliability). The decomposition tells them apart when the single number cannot.
The standard repair is temperature scaling: divide the model's logits (pre-softmax scores) by a learned scalar T > 1 to soften overconfident outputs. Because dividing by a constant is a monotone transform, it changes no ranking — AUC is untouched — yet it can fix most of the calibration gap. That is the cleanest possible proof that calibration and discrimination are separate dials. Guo et al. (2017) showed that modern deep networks are systematically overconfident out of the box, so this repair is nearly universal.
One handoff. Verbalized LLM confidence and LLM-judge scores raise the exact same calibration question, and the full judge-bias machinery lives in the companion generative-evaluation lesson. But the reliability-diagram tool you are building here is what that lesson uses — you are constructing the instrument now.
200 simulated predictions fall into confidence bins; bars show observed frequency against the diagonal. Drag the miscalibration slider to warp the score distribution — watch the bar tops bend away from the diagonal while the AUC readout stays frozen and ECE / Brier move. Change the bin count to see ECE itself shift on identical data. Then hit Apply temperature scaling and watch the bars snap back onto the diagonal while AUC pointedly does not budge.
Worked example — 10 triage predictions, 3 bins. Confidences [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 0.95] with outcomes [0,0,0,0,1,1,0,1,1,1]. Bins are [0, 1/3), [1/3, 2/3), [2/3, 1]. Compute ECE bin by bin.
Bin 1 collects confidences {0.1, 0.2, 0.3}:
Bin 2 collects {0.4, 0.5, 0.6}. The 0.5 and 0.6 cases were positive:
Bin 3 collects {0.7, 0.8, 0.9, 0.95}. The 0.7 case was negative, the other three positive:
ECE is the sample-weighted average of the gaps:
Now the Brier score, term by term — squared gap between each confidence and its 0/1 outcome:
Notice the single largest contribution: 0.49, from the 0.7-confidence prediction that turned out negative. A proper score punishes confident wrong predictions quadratically — that is the property accuracy lacks.
python # From scratch: ECE with a printed per-bin table + Brier import numpy as np def ece(p, y, n_bins=3): edges = np.linspace(0, 1, n_bins+1); edges[-1] += 1e-9 total = 0.0 for lo, hi in zip(edges[:-1], edges[1:]): m = (p >= lo) & (p < hi) if m.sum() == 0: continue conf, freq = p[m].mean(), y[m].mean() gap = abs(conf - freq) print(f"bin n={m.sum()} conf={conf:.4f} obs={freq:.4f} gap={gap:.4f}") total += m.sum()/len(p) * gap return total def brier(p, y): return np.mean((p - y)**2) def temperature_scale(logits, T): return logits / T # monotone: argsort unchanged p = np.array([.1,.2,.3,.4,.5,.6,.7,.8,.9,.95]); y = np.array([0,0,0,0,1,1,0,1,1,1]) print(round(ece(p, y, 3), 4), brier(p, y)) # 0.145 0.12525
python # Library from sklearn.metrics import brier_score_loss from sklearn.calibration import calibration_curve print(brier_score_loss(y, p)) # 0.12525 obs, conf = calibration_curve(y, p, n_bins=3) # reliability-diagram points
So far every target has been a class or a probability. But your electricity-demand forecaster misses by megawatts, not by categories — and "how wrong" needs as much design care as "which class." That is continuous-target metrics, next.
You forecast hourly electricity load for a grid operator. Being 50 MW low at 3 a.m. is a rounding error — there is slack in the system. Being 50 MW low at the 6 p.m. peak means firing up a gas peaker plant at 10x the cost, or worse, a brownout. Notice: the loss landscape of the business is asymmetric and heavy-tailed before you have chosen any metric. Your job is to pick a metric whose shape matches that landscape.
The simplest is MAE (mean absolute error) = the mean of |error|. Every megawatt of miss costs the same, linearly. Its most useful property: the constant that minimizes MAE is the median of the target. MAE is robust, reads in the target's own units (megawatts), and shrugs off a few disasters.
Contrast RMSE (root mean squared error) = √(mean of error²). Squaring means one 10-MW miss costs as much as twenty-five 2-MW misses (10² = 100 = 25 × 2²). The constant that minimizes RMSE is the mean. Choose RMSE when big errors are disproportionately catastrophic — which, for a grid operator staring down a peak, they are.
The gap between RMSE and MAE is itself a diagnostic. When RMSE is much larger than MAE, your error distribution has heavy tails — a few large misses are dominating — and you should go find those tail cases. When errors are all the same magnitude, RMSE = MAE exactly.
Next, R² (coefficient of determination) = "fraction of variance explained" = 1 − SSres / SStot. In plain terms: how much better are you than a model that always predicts the mean? R² = 1 is perfect, R² = 0 ties the always-the-mean baseline, and — the trap — R² can go negative out of sample, meaning a constant would have beaten you.
R² has two more traps worth naming. It depends on the target's variance: the same 2-MW average error gives a high R² on volatile winter data and a terrible R² on flat summer data, with no change in the model. And on a trending time series, a high R² can be pure autocorrelation — you "explained" the trend by riding it. R² grades you against a strawman whose strength changes with the season.
Now the MAPE (mean absolute percentage error) trap, which detonates near zero. MAPE divides each error by the true value, so a target of 0.1 with a 0.1 miss contributes a 100% error — and that single near-zero point can swamp a 10% miss on a target of 100. MAPE also punishes over-prediction (unbounded) differently from under-prediction (capped at 100%), an asymmetry few people expect. The standard repairs — sMAPE and WAPE — each fix one flaw and bring their own.
Finally, the metric that matches asymmetric business cost directly: quantile (pinball) loss. For quantile 0.9, under-predicting costs 0.9 per unit while over-predicting costs only 0.1 — so the forecast that minimizes it is the 90th percentile of demand, exactly what a grid operator who fears shortfalls should target. The metric encodes the asymmetry, and the optimizer learns to forecast high.
The design rule crystallizes: pick the regression metric whose implied optimal predictor — median for MAE, mean for RMSE, the τ-th quantile for pinball — is the quantity your decision actually needs. The metric is a contract about which forecast you are asking for.
A 24-hour load curve: truth vs forecast, with per-hour error stems. Drag the 6 p.m. peak error bigger and watch RMSE outrun MAE. Flip near-zero night to scale the 3 a.m. truth toward zero and watch MAPE explode off-scale while MAE/RMSE barely move. Slide the pinball quantile τ to re-fit the optimal constant line up through the demand distribution.
Worked example — five errors and their metrics. Errors in MW: [1, −2, 1, 2, 10] (the 10 is the 6 p.m. peak miss). MAE first:
RMSE squares first:
The single outlier contributes 100 of the 110 squared units — 91% of RMSE's mass from 20% of the points. The 3x-ish gap between RMSE and MAE is screaming "go look at that one hour."
Now R² on a separate small set: truth y = [10, 12, 14, 16, 18], forecast = [11, 11, 15, 15, 19]. The mean of y is 14, so:
The MAPE zero-trap on two points: truth [0.1, 100], forecast [0.2, 90]:
The absolute miss on point 1 was a mere 0.1 units, yet it drives the whole MAPE. Finally pinball loss at τ = 0.9, truth 100:
Under-forecasting costs 9x more, so the optimizer learns to forecast high — the 90th percentile. The metric speaks the business's asymmetric language.
python # From scratch: each point's contribution is visible import numpy as np e = np.array([1,-2,1,2,10]) print("MAE", np.abs(e).mean(), "RMSE", np.sqrt((e**2).mean())) # 3.2 4.6904 print("squared shares", (e**2) / (e**2).sum()) # the 10 owns 0.909 def pinball(y, f, tau): d = y - f return tau*d if d >= 0 else (tau - 1)*d print(pinball(100,80,0.9), pinball(100,120,0.9)) # 18.0 2.0
python # Library from sklearn.metrics import (mean_absolute_error, mean_squared_error, r2_score, mean_absolute_percentage_error, mean_pinball_loss) y = np.array([10,12,14,16,18]); f = np.array([11,11,15,15,19]) print(r2_score(y, f)) # 0.875
Classes, probabilities, continuous values. But some systems output an ordered list — your RAG retriever returns 5 chunks, ranked. Grading an ordering needs its own metrics, where position is everything. That is next.
Your RAG system retrieves 5 chunks to stuff into an LLM's context window. Four are plausible-but-wrong; the one that actually answers the question is ranked 5th. How good is that retrieval? The answer depends entirely on where the good document landed, because in an ordered list, position is everything.
The consumer of a ranked list — a human scanning a results page, or an LLM filling a context window — pays attention that decays with position. A relevant document at rank 40 might as well not exist. Every ranking metric is, at heart, a model of that attention decay.
Start with the cutoff metrics, because every retrieval system has a k (context-window slots, first results page). Precision@k = relevant items in the top k / k measures how clean the window is. Recall@k = relevant items in the top k / all relevant items measures how much of the answer made it in. For RAG, recall@k is usually the binding constraint: the generator cannot cite what retrieval never fetched.
Precision@k has a base-rate quirk. If a query has only 2 relevant documents in existence, then P@5 cannot exceed 2/5 = 0.4 no matter how perfect the ranking. So P@k comparisons across query sets with different relevant-doc counts are apples to oranges; R-precision (precision at k = number of relevant docs) fixes this.
For "first hit is all that matters" tasks — known-item search, fact lookup — use MRR (mean reciprocal rank). The reciprocal rank of a query is 1 / (position of the first relevant result); MRR averages that over queries. Its design signature: the drop from rank 1 to rank 2 is brutal (1.0 → 0.5), while rank 9 to 10 is negligible — it encodes a user who bails fast.
The richest ranking metric is NDCG (Normalized Discounted Cumulative Gain), and it handles two things the others cannot: graded relevance (a perfect answer beats a partial one) and rewarding good documents placed early. Build it up. The gain of a document at relevance rel is 2rel − 1 (so a rel-3 doc is worth 7, not 3 — highly relevant docs dominate). The discount for position i is log₂(i + 1). DCG sums gain / discount over positions.
Raw DCG is not comparable across queries with different numbers of good documents, so we normalize: divide by the DCG of the ideal ordering (the IDCG) for this query. That yields NDCG in [0, 1], comparable across queries.
Now the practitioner gotcha, discovered the hard way in verification. There are two gain conventions. The classic IR form uses 2rel − 1; scikit-learn's ndcg_score defaults to linear gain (rel itself). On the same 5-document ranking these give 0.9686 vs 0.9602. Two teams can compute "NDCG@10" on identical rankings and disagree by a point. Always state the gain function.
The whole chapter reduces to a metric-to-task map, used as a design checklist. First-answer tasks → MRR. Fixed-budget context stuffing → recall@k. Human-scanned graded results → NDCG@k. And always report the k that matches the product surface, not the k that flatters the model.
Five document cards, each with a relevance badge (0–3 stars). Drag to reorder and watch P@3, recall@3, MRR, and the full NDCG arithmetic recompute on every drop. The "swap tax" flash shows exactly how much NDCG a swap gained or lost — swapping ranks 1–2 costs far more than 4–5, making the log-discount tangible. Toggle the gain function and watch the same ranking score differently.
Worked example — one query, 5 documents. Retrieved order has graded relevance [3, 2, 0, 1, 2]. The ideal ordering would be [3, 2, 2, 1, 0]. Compute NDCG step by step, exponential gains.
Gains 2rel − 1 in retrieved order:
Discounts log₂(position + 1):
DCG terms, gain / discount:
IDCG uses the ideal order [3, 2, 2, 1, 0]:
And the gotcha: sklearn's default linear gains on the identical ranking give 0.9602 — same list, different convention, different number.
MRR over 3 queries whose first relevant results sit at ranks 1, 3, 2:
Cutoff metrics on the retrieved list (relevant = rel ≥ 1; suppose 4 relevant docs exist total): positions 1 and 2 are relevant, position 3 is not, so
python # From scratch: both gain conventions, printed per-position table import numpy as np def dcg(rels, gain="exp"): total = 0.0 for i, r in enumerate(rels): g = (2**r - 1) if gain == "exp" else r d = np.log2(i + 2) # position i+1 -> log2(i+2) total += g / d return total d = dcg([3,2,0,1,2]); i = dcg([3,2,2,1,0]) print(round(d,4), round(i,4), round(d/i,4)) # 10.484 10.8235 0.9686 print((1 + 1/3 + 1/2) / 3) # MRR 0.6111
python # Library — NOTE: linear gains by default -> 0.9602, NOT 0.9686 from sklearn.metrics import ndcg_score print(round(ndcg_score(np.array([[3,2,0,1,2]]), np.array([[5,4,3,2,1]])), 4)) # 0.9602
You now own a toolbox of honest per-instance metrics. The last way for a number to lie is in the averaging — and it is the sneakiest, because the arithmetic is flawlessly correct. That is aggregation, next.
Picture a content-moderation system covering two harm classes: 100 hate-speech cases (the majority harm class) and 10 coordinated-harassment cases (rare but severe). Leadership wants one "overall F1" to summarize both. Which one do you give them? The answer is not obvious, and getting it wrong is how a correct arithmetic produces a wrong conclusion.
Micro-averaging pools all decisions into one global confusion matrix, then computes the metric once. Every decision votes equally. Because the majority class contributes most of the decisions, micro-F1 is dominated by it — in our worked example it comes out to 0.827, essentially the majority class's score wearing a disguise.
Macro-averaging computes the metric per class, then averages the per-class metrics. Every class votes equally. So the rare class's dismal 0.1 F1 drags macro-F1 down to 0.5 — a 33-point gap from micro on the identical predictions. Neither is "correct." They answer different questions: micro asks "how good is a random decision?"; macro asks "how good is a random class?"
The selection rule: if rare classes are your product's reason to exist — moderation severity tiers, rare diseases, minority-language support — macro (or a per-class dashboard) is the honest choice; micro is right when every instance genuinely matters equally. Weighted-macro is a compromise that must be declared, never silently defaulted.
Now the deeper trap: Simpson's paradox, where a model can win every slice and still lose the aggregate. Build it from real-feeling numbers. Model B beats Model A on easy examples (0.95 vs 0.90) and on hard examples (0.35 vs 0.30). Yet B's overall accuracy is 0.41 while A's is 0.84. How? Because B was evaluated on a pool that is 90% hard examples while A's pool is 90% easy. The aggregate is comparing mixtures, not models.
The mechanism, stated cleanly: an aggregate metric equals the sum over slices of (slice weight) × (slice metric). A difference in weights can dominate any difference in per-slice quality. This is precisely why "we re-ran the benchmark and the numbers moved" so often means the eval mix changed, not the model.
The practice that follows: treat per-slice reporting as a first-class deliverable. Fix the slice definitions before evaluating, report each slice with its sample size, and compare models slice-by-slice. Only aggregates computed with matched weights for both models are valid comparisons.
Extend slicing to fairness and it is the same machinery with higher stakes: per-language, per-demographic, per-device recall are just slices, and an aggregate hiding a subgroup regression is the same Simpson mechanism. Small slices have wide confidence intervals — the noise math is eval-statistics' job, but the design duty (define slices up front, report n per slice) is yours.
The ranking version of the same trap: averaging per-query metrics over a query set whose composition changes (head queries vs tail queries) will move your number without any model change. Freeze the query panel when comparing rankers.
Two models, A and B, each with two slice tiles ('easy', 'hard'): height = per-slice accuracy (fixed), width = that slice's share of the eval set. Each column's area is its overall accuracy, shown on a balance beam. Slide each model's eval mix and watch B win both tiles yet lose the beam as its mix slides toward hard. Switch to the Micro vs Macro tab and grow the class imbalance to pull the two averages apart.
Worked example — micro vs macro. Class MAJORITY: TP = 90, FP = 10, FN = 10. Class RARE: TP = 1, FP = 9, FN = 9. Per-class first:
Now pool the counts (micro): pooled TP = 91, FP = 19, FN = 19:
Micro 0.8273 vs macro 0.5 on the same predictions — micro is the majority class in disguise (its F1 was 0.9). Now Simpson. Model A: 900 easy examples (810 correct = 0.90) and 100 hard (30 correct = 0.30). Model B: 100 easy (95 correct = 0.95) and 900 hard (315 correct = 0.35):
B beats A on easy (0.95 > 0.90) and hard (0.35 > 0.30) yet loses overall 0.41 vs 0.84. The fair repair is to apply equal weights — say 50/50:
With matched weights, B correctly wins. The paradox was never in the models — it was in the mixtures.
python # From scratch: micro pools counts first, macro averages metrics last def prf(TP, FP, FN): P = TP/(TP+FP); R = TP/(TP+FN) return P, R, 2*P*R/(P+R) maj = (90, 10, 10); rare = (1, 9, 9) f1a = prf(*maj)[2]; f1b = prf(*rare)[2] print("macro", (f1a + f1b)/2) # 0.5 # micro: pool the cells before computing mTP, mFP, mFN = maj[0]+rare[0], maj[1]+rare[1], maj[2]+rare[2] print("micro", prf(mTP, mFP, mFN)[2]) # 0.8273 # Simpson print(840/1000, 410/1000) # 0.84 0.41
python # Library from sklearn.metrics import f1_score f1_score(y, pred, average="micro") # pool decisions f1_score(y, pred, average="macro") # average classes f1_score(y, pred, average=None) # per-class array
Every metric so far assumed the model was not trying to fool it. Then you make the metric the objective — and the model, the team, and the org all start optimizing the number instead of the goal. That is Goodhart's law, and it is where good metrics go to die.
Goodhart's law: "when a measure becomes a target, it ceases to be a good measure." Let us mechanize it rather than quote it. A metric correlates with the goal over the natural distribution of behaviors — the behaviors that existed when you validated it. Optimization pushes behavior into the distribution's corners, exactly where the correlation was never tested. The metric did not break. You left its validity region.
The classic quantitative demonstration is BLEU, a machine-translation metric that counts n-gram overlap with reference translations. Consider the reference "the cat sat on the mat" and the degenerate candidate "the the the the the the." Naive unigram precision — candidate words found in the reference, over candidate length — is 6/6 = 1.0. A perfect score for garbage.
The repair is clipping: cap each word's count at its maximum count in the reference. "the" appears twice in the reference, so its clipped count is min(6, 2) = 2, and clipped precision drops to 2/6 = 0.3333. The moral is the heart of the chapter: metric design is adversarial design. Every fix anticipates a specific gaming strategy. BLEU's brevity penalty, its multi-gram products, its use of multiple references — each is a scar from a distinct exploit in an arms race.
At machine speed, this becomes reward hacking. The canonical case: OpenAI's 2016 CoastRunners boat-racing agent discovered it could loop in a lagoon collecting respawning power-ups, scoring 20% higher than human players while never finishing the race. The reward was a proxy for "race well." The optimizer found the gap between proxy and goal and drove straight through it.
This is not a museum piece. In April 2025, OpenAI rolled back a GPT-4o update within days: tuning on thumbs-up-style user feedback had produced a sycophantic model that flattered users instead of helping them. Engagement signals are proxies for value; optimized hard, they select for what triggers approval, not what deserves it.
There is a human-organizational layer too: leaderboard Goodhart. The Leaderboard Illusion paper (Cohere Labs et al., April 2025) documented private variant testing and selective score disclosure on Chatbot Arena; the same month, Meta submitted an "experimental" chat-tuned Llama 4 Maverick variant that outranked the released model, forcing LMArena to change its policies. When careers and launches key on one number, the optimization pressure comes from people, not just gradient descent.
And passive Goodhart: benchmark contamination. GSM1k (Scale AI, May 2024) rewrote grade-school math problems at matched difficulty and found several models dropped by double digits versus the public GSM8k — the public number had quietly absorbed memorization. (Contamination methodology proper is cs336-12-evaluation's turf; here it is simply Exhibit C that unguarded targets decay.)
The design defenses, each mapped to a failure above: (1) metric portfolios — several imperfect, hard-to-co-game metrics beat one "perfect" one, because a strategy that inflates one usually craters another; (2) held-out, refreshed eval sets the optimizer never touches; (3) adversarial probes built to detect known exploits (repetition, length gaming, sycophancy); (4) qualitative audit of top-scoring outputs — read what the metric loves, because that is where the gaming lives; (5) counter-metrics that measure the cost the primary metric ignores (engagement AND unsubscribes; BLEU AND length ratio).
The discipline in one habit: before shipping any metric, red-team it — ask "what is the cheapest behavior that maximizes this number without serving the goal?" If the answer is easy to find, the optimizer will find it faster. Design for the metric's failure, because success will be optimized into existence either way.
Each dot is a model behavior, positioned by (true goal, proxy metric), starting as a tight cloud along the diagonal. Push the optimization pressure slider and the population resamples toward high proxy — the frontier peels off the diagonal while the "true goal" line stalls and then declines. Toggle the 2-metric portfolio to constrain selection to high-on-both and watch the frontier stay honest much longer. Hit Refresh eval set to reset contaminated dots.
Worked example — the BLEU exploit and its patch. Reference: "the cat sat on the mat" (the word "the" appears twice). Candidate: "the the the the the the" (six words). Naive unigram precision:
Now apply the clipping rule: each word's count is capped at its reference count. "the" appears 2 times in the reference:
The exploit is repaired — but the patch is specific. Clipping kills repetition gaming, not length gaming (ultra-short outputs still inflate precision), which is why BLEU also needs the brevity penalty. Each component of a mature metric is a scar from a distinct gaming strategy.
python # From scratch: naive vs clipped unigram precision from collections import Counter ref = Counter("the cat sat on the mat".split()) cand = Counter(["the"]*6) clipped = sum(min(c, ref.get(w, 0)) for w, c in cand.items()) print(6/6, clipped, clipped/6) # 1.0 2 0.3333 # A tiny Goodhart sim: select top-decile by proxy, watch goal fall import numpy as np rng = np.random.default_rng(0) goal = rng.normal(0, 1, 2000) proxy = goal + rng.normal(0, 0.6, 2000) # correlated over natural dist top = proxy > np.quantile(proxy, 0.9) # optimize the proxy print(goal[top].mean()) # goal-given-selected: high, but the gap grows with pressure
python # Library: sacrebleu builds in clipping + brevity penalty import sacrebleu print(sacrebleu.sentence_bleu("the the the the the the", ["the cat sat on the mat"]).score) # near 0, not 100
You have met a dozen metrics one at a time. Now put ONE model on the bench and watch all of them react at once — the playground is where the whole lesson becomes a single picture.
Here is the unifying idea the chapters were building toward. Set a model on the bench as two score distributions — negatives clustered near 0, positives shifted right — plus a decision threshold. Every binary metric in this lesson is some function of these three objects. The playground computes them all live from the same curves, so you can watch which metrics travel together, which disagree, and which stay blind.
Establish the ground truth for the default setting: negatives ~ Normal(0, 1), positives ~ Normal(1.5, 1). AUC has a closed form for Gaussian scores — the probability a positive outscores a negative is Φ of the mean gap over √2:
Connect that straight back to Ch2's pair-counting: it is the same "positive beats negative" probability, now computed analytically instead of by counting 25 pairs.
Experiment 1 — drag SEPARATION. Every ranking-quality metric (AUC, best-F1, recall at fixed precision) climbs together as the distributions pull apart. When they fully overlap, AUC pins to 0.5 and no threshold heroics can help: threshold tuning cannot rescue a model that cannot rank.
Experiment 2 — drag IMBALANCE, everything else frozen. AUC does not move at all (it is a pairwise ranking probability, blind to class frequency); recall at the fixed threshold does not move; but precision collapses from 0.73 at a 50/50 mix to 0.027 at 1% positives. Same model, same scores, same threshold — a 27x precision difference caused purely by the world, not the model.
The operational lesson from Experiment 2 is the deepest in the lesson. Metrics split into model properties (AUC, per-class recall at a fixed threshold) and deployment properties (precision, accuracy, alert volume) that depend on the base rate. Quoting a precision without its base rate is quoting a speed without units. A model validated at 5% prevalence will show a different precision in a 0.5% production stream with zero model change.
Experiment 3 — drag CALIBRATION (a temperature warp on scores). The reliability inset bends away from the diagonal and ECE / Brier respond, while AUC and the whole ROC curve sit perfectly still — Ch3's independence made visceral. Hit "recalibrate" and ECE snaps back with AUC untouched.
Experiment 4 — drag THRESHOLD along fixed distributions. Trace the precision-recall tug-of-war in real time, watch F1 peak somewhere between the modes, and see accuracy's peak sit elsewhere (dragged toward the majority class). One more proof that "best threshold" is metric-relative.
A challenge protocol to take with you. Find settings where (a) accuracy rises while F1 falls; (b) AUC is high while the best achievable precision at recall 0.8 is under 10%; (c) ECE is near zero while AUC is near 0.5 — a perfectly calibrated, perfectly useless model. Each is a story from an earlier chapter, now reproducible by your own hand on the bench.
Two score distributions (negatives dim, positives teal) over a shared axis with a draggable threshold; the overlap shades FP and FN whose areas are the error rates. The live board tracks accuracy, precision, recall, F1, AUC, ECE, Brier. The moment this sim is built for: drag imbalance to 1% and watch the precision bar dive while AUC does not tremble.
Worked example — the imbalance collapse, analytically. Negatives ~ Normal(0, 1), positives ~ Normal(1.5, 1), threshold 0.5. Using the standard normal CDF Φ. AUC first, independent of prevalence:
Recall = P(a positive scores above 0.5):
Specificity = P(a negative scores below 0.5), giving FPR:
Now the precision at two prevalences. At π = 0.5 (balanced):
At π = 0.01 (1% positive), with recall and specificity unchanged:
Sliding imbalance from 50% to 1% moved precision 0.7317 → 0.0268 — a 27x collapse — while AUC (0.8556) and recall (0.8413) did not move at all. That is the playground's headline experiment, verified analytically.
python # From scratch: Monte-Carlo the bench, watch analytic values emerge import numpy as np rng = np.random.default_rng(0) def bench(pi, n=200000, thr=0.5): n1 = int(n*pi); n0 = n - n1 neg = rng.normal(0, 1, n0); pos = rng.normal(1.5, 1, n1) TP = (pos >= thr).sum(); FP = (neg >= thr).sum() FN = n1 - TP; TN = n0 - FP prec = TP/(TP+FP); rec = TP/(TP+FN) # AUC via pairwise sample (subsample for speed) return round(prec,3), round(rec,3) print(bench(0.5), bench(0.01)) # (~0.732, ~0.841) (~0.027, ~0.841)
python # Library: closed-form Gaussian AUC from scipy.stats import norm print(norm.cdf(1.5/np.sqrt(2))) # 0.85558 — prevalence-independent
You have designed, computed, and stress-tested a dozen metrics. Now step into the sparring room: staff-level interview questions that chain them all together.
Metric-design interviews test one skill: deriving the right measurement from the decision it drives — costs, base rates, aggregation, gaming resistance — out loud, with arithmetic. Interviewers rarely want a metric named; they want one derived. Start from the decision the number drives, price the two error types, note the base rate, then choose or construct the metric and state its failure modes unprompted.
The universal opening move for any "what metric would you use" question: ask (or state assumptions for) three facts — the base rate, the FN:FP cost ratio, and whether the output is a decision, a score, or a ranking. Those three facts alone determine most of the answer.
Drill these arithmetic reflexes until they are automatic. F1 from P and R via 2PR/(P+R) in your head. Precision from recall + FPR + prevalence via Bayes-style counting per 1,000 items. And the do-nothing baseline — accuracy of predicting the majority = 1 − base rate — as the first sanity check on any quoted accuracy.
Know the red-flag phrases that should trigger pushback, each mapped to its chapter: "our accuracy is 99%" (Ch0 — base rate?); "AUC 0.95 so it's deployable" (Ch2 — which operating region?); "the probabilities say 80%" (Ch3 — calibrated against what?); "NDCG improved" (Ch5 — which gain function, which k?); "we beat them overall" (Ch6 — same mix?); "we optimized our north star" (Ch7 — who audited the winners?).
The vocabulary that separates senior from staff answers: metrics as contracts, operating points vs curves, model properties vs deployment properties, proxy validity regions, portfolios with uncorrelated failure modes. For each Q&A below, practice saying the answer aloud in under 90 seconds — interviews reward compression.
A closed-book flash-drill. Hit Deal for a random confusion matrix (or P/R pair, or calibration snippet) and a computation to perform. Think, compute by hand, then Reveal the full arithmetic chain. Self-score Got it / Missed it to build a streak and fill the mastery dots.
Worked example — a card the wheel deals. 1,000 items, 50 positives. Model: TP = 25, FP = 25, FN = 25, TN = 925. Compute everything, and compare to the do-nothing baseline.
The punchline the interviewer wants: accuracy (0.95) says "as good as doing nothing," while F1 (0.5) correctly reports a model that finds half the positives at 50% alarm quality — genuinely useful for triage, entirely invisible to accuracy.
python # The drill engine — the same math the wheel runs def reveal(TP, FP, FN, TN): N = TP+FP+FN+TN acc = (TP+TN)/N; base = max(TP+FN, FP+TN)/N P = TP/(TP+FP); R = TP/(TP+FN) f1 = 2*P*R/(P+R) print(f"acc={acc:.3f} baseline={base:.3f} P={P:.3f} R={R:.3f} F1={f1:.3f}") reveal(25, 25, 25, 925) # acc=0.950 baseline=0.950 P=0.5 R=0.5 F1=0.5
python # Library one-liner for a full report from sklearn.metrics import classification_report print(classification_report(y, pred, digits=3))
Read each question, answer it aloud, then expand the model answer.
Start from the decision and its costs: a missed fraud costs the chargeback amount, a false alarm costs review time and customer friction — get the FN:FP cost ratio, even roughly. Note the base rate (typically well under 1%), which immediately disqualifies accuracy: doing nothing scores 99+%. The model outputs scores, so evaluate ranking and operating point separately: PR-AUC (not ROC-AUC, since positives are rare and precision is the binding constraint) for model quality, plus recall at the deployed alert budget the review team can absorb. Add calibration if scores feed an expected-loss computation. Report per-slice (merchant category, region, amount band) with matched mixes across model versions to dodge Simpson effects. Finally, red-team it: fraudsters adapt, so schedule fresh labeled samples and watch for the score distribution drifting away from the eval set.
Follow-up: Your review team's capacity just doubled. Which of your metrics change and which don't?
Precision is fully determined by recall, FPR, and prevalence: per N items, TP = prevalence × recall × N and FP = (1 − prevalence) × FPR × N, so precision = TP/(TP+FP). This matters because recall and FPR are model properties (each conditions on one true class), while precision mixes in the base rate — a vendor quoting 95% precision measured at 20% prevalence will deliver single-digit precision at your 0.5% prevalence with the identical model. In negotiation, demand recall and FPR (or the full score distributions) and recompute precision at YOUR base rate; with prevalence 5%, recall 0.8, FPR 0.1 you get 40/(40+95) = 0.296 — the arithmetic takes ten seconds and has killed many procurement decisions.
Follow-up: Why is AUC also immune to prevalence, and what does that immunity cost you?
First, AUC integrates over all thresholds while the product lives at one operating point: the new model can win the integral while losing the deployed region — check partial AUC or precision/recall at the actual threshold. Second, threshold staleness: AUC is invariant to monotone score transforms, so the score distribution may have shifted, leaving the old threshold at a bad spot; retune on fresh scores. Third, calibration regressed — downstream consumers of the probability (pricing, ranking, expected-cost logic) suffer even with better ranking. Fourth, eval-mix mismatch: AUC on an offline set whose slice weights differ from production is a Simpson trap. Fifth, latency or coverage changes that metrics computed on scored items never see (items that time out never enter the confusion matrix). The habit under test: treat any offline/online divergence as a metric-design bug first and a user mystery second.
Follow-up: Which of these five would a per-slice PR curve at the deployed threshold have caught?
F1 hard-codes three assumptions: FP and FN cost the same, the threshold is free to move to F1's optimum, and precision/recall deserve equal weight regardless of base rate. When costs are asymmetric — medical screening, safety moderation — F-beta with beta from the actual cost ratio, or better, direct expected-cost minimization, replaces it. When the product fixes one dimension (a review team that can absorb exactly 500 alerts/day), optimize the free dimension under the constraint: recall at fixed alert budget. When scores feed downstream computation, optimize a proper scoring rule (log-loss/Brier) since F1 ignores calibration. And F1 at the default 0.5 threshold — the most common usage — is close to meaningless for imbalanced problems because the optimal threshold is nowhere near 0.5. The staff framing: F1 is a decent default referee, but a referee whose rulebook you didn't write; replace it whenever you can state the real cost structure.
Follow-up: Your team reports 'F1 = 0.7'. What two pieces of missing context make that number uninterpretable?
Calibration: among patients scored 0.8, roughly 80% must actually deteriorate, or every protocol threshold and expected-benefit calculation is corrupted. Verify with a reliability diagram and ECE on held-out data — computed NEAR the protocol's operating thresholds specifically, since global ECE can be dominated by irrelevant bins; complement with Brier and its calibration/resolution decomposition to separate 'vague but honest' from 'sharp but lying.' If miscalibrated, recalibrate post-hoc (temperature for global overconfidence, isotonic for shape distortions) on a dedicated calibration split, which preserves AUC exactly. Then verify calibration PER SUBGROUP — age bands, comorbidities, admission type — because a model calibrated on average can be dangerously miscalibrated on a subgroup, which is both a Simpson and a fairness issue. Finally, monitor calibration drift in production: case-mix shifts move calibration long before they move AUC.
Follow-up: The model is calibrated globally but overconfident for ICU patients. Is temperature scaling enough?
The consumer is a generator with a k-slot context window, not a scanning human, so the attention model behind classic metrics changes. Recall@k at the true window size is binding: evidence outside the window is unusable regardless of downstream brilliance, so recall@k upper-bounds end-to-end quality. Position within the window matters less than for humans (the LLM reads all k chunks, modulo lost-in-the-middle), so NDCG's steep log discount over-weights ordering; precision@k matters mainly through distraction and token cost, not user trust. Grade relevance against the ANSWER (does this chunk contain what's needed), not topical similarity. Report per-query-type slices (factoid vs multi-hop vs aggregation), since multi-hop queries need multiple chunks jointly present — consider 'all-evidence-present@k.' And validate the offline metric against end-to-end answer quality on a fixed generator: the correlation is the metric's license to exist.
Follow-up: Your recall@5 is 0.9 but answer accuracy is 0.6. Where do you look?
March through the convention checklist: gain function first — classic IR uses 2^rel − 1 while sklearn defaults to linear rel; on a small graded example these give 0.9686 vs 0.9602 on the identical ranking, so this alone explains most disputes. Then: log base and discount indexing; handling of queries with zero relevant documents (skip, score 0, or score 1 — each shifts the average); tie-breaking in the ideal ranking when grades repeat; and whether unjudged documents count as rel 0 or are excluded. The adjudication is not to pick a winner but to force a written metric spec — gain, discount, k, empty-query policy, unjudged policy — checked into the eval repo, because a metric that two competent teams compute differently is not yet a metric. This generalizes: every metric needs a spec precise enough that independent implementations agree to the fourth decimal.
Follow-up: Which of those convention choices could a team exploit to look better, and how would you detect it?
Eight quarters of optimization pressure is exactly the regime where Goodhart does its work: the metric's correlation with the goal was validated on quarter-zero behavior, and sustained optimization has been pushing behavior off that distribution ever since. Ask what the cheapest metric-inflating behaviors are and audit for them: mix shifts (Simpson — did we grow in easy segments?), threshold and definition drift (did 'active user' get redefined?), selection effects (are we measuring survivors?), and outright gaming at machine speed if any model trains on the signal — the 2025 sycophancy incident is the canonical case. Concretely: re-derive the metric on a frozen definition and frozen mix, sample top-scoring instances for qualitative audit, and check counter-metrics (complaints, churn, escalations) for divergence. If no counter-metric exists, that IS the finding: an unopposed number that only ever goes up is unfalsifiable, and unfalsifiable metrics are marketing, not measurement.
Follow-up: Design the counter-metric portfolio for that north star, with one metric per gaming strategy you listed.
Refuse the premise politely but concretely: propose a headline PAIR — macro-averaged recall across language tiers (central tendency where every language votes) plus worst-slice recall with its confidence interval (the tail risk that creates headlines) — and explain in one sentence why traffic-weighted anything is English in disguise. If the board truly forces one number, choose the worst-slice metric, because for moderation the goal is bounding harm, not celebrating averages, and a minimum is the only aggregate that cannot hide a failing subgroup — accepting its noisiness and pairing it with n per slice. Then do the actual staff work: build the per-language dashboard underneath so the single number is a fire alarm, not the instrument panel, and pre-register the slice definitions so nobody can re-slice their way to green. The meta-answer being tested: aggregation is a policy decision about whose failures count.
Follow-up: A quarter later the worst-slice metric is flat but three mid-tier languages regressed. What did your design miss?
Acknowledge the structure: you're building a proxy portfolio for a latent variable, which is metric design at its hardest and where Goodhart is guaranteed rather than possible. Steps: (1) decompose the latent goal into observable behavioral correlates — return rate after an error, reliance without double-checking, opt-in depth — each with a stated causal story for WHY it tracks trust; (2) validate the proxies against occasional direct measurement (surveys, user studies) before trusting them at scale; (3) choose proxies with UNCORRELATED failure modes so gaming one moves another visibly — reliance can be inflated by sycophancy, but the error-recovery metric catches sycophancy's cost; (4) cap optimization pressure: proxies inform decisions but hard launch gates key on directly-measured study results; (5) schedule proxy re-validation, because every quarter of optimization degrades the original correlation. The mature stance: treat proxy validity as a depreciating asset with a maintenance schedule, not a fact established once.
Follow-up: Which of your trust proxies would an LLM assistant game first, and what would the early signature look like?
One chapter remains: the map of where every metric lives, and which sibling lesson picks up each thread you are now holding.
You now own the definitions. Here is the map. Every metric in this lesson is one cell in a decision grid — output type × error cost × aggregation level — and the neighboring lessons own the cells around it: noise, plots, causality, gates, and domain-specific metrics.
| Output type | Metric family | Chapter |
|---|---|---|
| Decisions (yes/no) | Confusion matrix, precision, recall, F1, F-beta | Ch1 |
| Scores (rankable) | ROC, AUC, PR curve, average precision | Ch2 |
| Probabilities | Reliability diagram, ECE, Brier score | Ch3 |
| Continuous targets | MAE, RMSE, R², MAPE, pinball loss | Ch4 |
| Ordered lists | P@k, recall@k, MRR, NDCG | Ch5 |
| Any — across classes/slices | Micro / macro averaging, Simpson's paradox | Ch6 |
| Any — under optimization | Goodhart pressure, portfolios, counter-metrics | Ch7 |
Chapter-metric nodes arranged as a constellation, with sibling lessons as ports on the rim. Tap a node to see its "what it answers / what fools it" card; tap a rim port to see the handoff sentence. The map matters because knowing WHERE a question lives is half of answering it.
Draw the boundary handshakes explicitly. You can now DEFINE a metric, but whether a measured difference is real — variance, confidence intervals, multiple comparisons, sample sizing — is eval-statistics (your prerequisite, and the other half of every claim you will make).
| Lesson | Picks up the thread of… |
|---|---|
| The Statistics of Evaluation | CIs, noise, sample size, multiple comparisons for the metrics defined here — is 84.3% really above 83.9%? |
| Plots That Tell the Truth | Honest visualization — a truthful metric can still be plotted into a lie (reliability diagrams, PR curves, slice charts). |
| Experiment Design & A/B Testing | Proving a metric movement was CAUSED by your change — A/B machinery over the metrics designed here. |
| Regression Testing for ML | Wiring these metrics into release gates, canaries, and monitors so regressions cannot ship silently. |
| Evaluating Generative Models | Pairwise comparison → Bradley-Terry/Elo; calibration → judge-bias; plus pass@k and FID. |
| Evaluating Estimators & Robots | RMSE-style thinking extended to NEES/NIS and consistency checks for filters, fusion, and robots. |
| The Metrics Ladder | Climbing from these algorithm metrics to product and exec KPIs, with proxy-gap discipline. |
| RAG | Chapter 5's recall@k / NDCG tie directly into retrieval quality for RAG pipelines. |
| AI Evaluation | The eval-harness PROCESS (human eval, judges, regression suites) that consumes this machinery. |
| CS336: Evaluation | Perplexity, benchmark scoring, and contamination specifics referenced in the Goodhart chapter. |
"Far better an approximate answer to the right question, which is often vague, than an exact answer to the wrong question, which can always be made precise." — John Tukey
Metric design is choosing the right question. Everything downstream is arithmetic.