Plots That Tell
the Truth

Chapter 1: Distributions First — Bins, Bandwidths, and the ECDF

Before you ever compare two models with one number each, look at the full distribution of what they do. And beware: the tool you reach for to look — the histogram — has a hidden knob that can manufacture a story or erase one.

Take a concrete case. Your speech-recognition service records the latency of every request. The mean latency is 34 ms. But the mean is one number describing 200 numbers. Behind that 34 ms could hide a tight bell (every request is about 34 ms), a two-humped mix (cache hits at 15 ms, cache misses at 55 ms), or a long tail (most requests are fast but 2% are catastrophic 400 ms). Each shape demands a completely different engineering fix, and the mean cannot tell them apart.

The histogram is the default tool for seeing shape: chop the value range into bins, count how many points fall in each, draw bars. Its hidden argument is the bin width. Take the same 200 latencies. Five wide bins show one smooth hump — the bimodality is erased. Twenty medium bins show two clear modes — cache hit near 15 ms, cache miss near 55 ms. Two hundred tiny bins show spiky noise, every bin holding 0 or 1 points. Three bin widths, three incompatible stories, one dataset.

Rules of thumb exist precisely because eyeballing fails. Sturges' rule sets the bin count to 1 + log₂n — good for small, roughly-normal data, but it only looks at n, not the spread. Freedman-Diaconis sets the bin width to 2 × IQR / n^1/3 — robust to outliers because it uses the interquartile range (IQR) rather than the full range, which a single 400 ms outlier would blow up. We will work the FD formula by hand below. Treat both as starting points, never as truth: always wiggle the width and check the story is stable.

Kernel density estimation (KDE) replaces hard bins with a smooth bump placed on each point, then sums the bumps into a curve. It looks more sophisticated, but it just relocates the knob: instead of bin width you now tune bandwidth. Too small and the curve is spaghetti; too large and it collapses to one bland hump that hides the bimodality exactly like the 5-bin histogram did. Silverman's rule gives a default, and the same stability test applies.

Now the hero of this chapter: the ECDF, the empirical cumulative distribution function. Sort the data, then plot — for each value on the x-axis — the fraction of points at or below it on the y-axis. There is no bin width, no bandwidth, no tuning parameter of any kind. Every single data point is visible as one step in a staircase. You can read the median, any tail percentile (p95, p99), and the fraction of requests below any SLO threshold directly off the curve. Two models overlay as two clean step curves — no alpha-blended histogram mush.

Learn to read the ECDF's shape. A steep region means many points are packed there — that is a mode. A flat plateau means a gap between clusters where almost no points live — that flat shelf in the middle of the staircase is the unmistakable signature of bimodality. A long shallow right tail means a heavy tail of rare large values. Our cache-hit / cache-miss latencies show a clear plateau between roughly 25 and 45 ms, where the two clusters have no overlap.

What about box plots? A box plot compresses the whole distribution to five numbers (roughly min, Q1, median, Q3, max), and that compression is exactly why it is the classic bimodality-hider: a two-humped distribution and a flat uniform one can produce nearly identical boxes. Violin plots fix this by drawing the KDE silhouette on each side — but they inherit KDE's bandwidth knob. A practical hierarchy for eval reports: ECDF when you need precision or want to overlay two systems; violin when you need to eyeball many groups at once; box plot only when you have dozens of groups and need compactness — and say so in the caption.

Close the loop to evaluation. Latency SLOs, per-example loss distributions, LLM-judge score distributions, reward distributions in RL — in every case, plot the distribution before trusting any mean-based comparison. A mean regression of +2 ms could be "everything got slightly slower" (probably harmless noise) or "a new 1% tail of 500 ms requests" (an incident). The mean is identical in both; only the distribution view distinguishes them.

Drag the bin slider below. One fixed dataset of 200 bimodal latencies never changes, but the histogram's story flips from "one hump" to "two modes" to "noise" as you slide. The ECDF panel beneath it never changes — watch its plateau stay put no matter what the histogram claims. Drag the threshold line to read off "X% of requests ≤ T ms."

Worked example: FD width and reading an ECDF

Take eight ASR request latencies in ms: 12, 15, 18, 22, 30, 45, 80, 210. We will compute (a) the Freedman-Diaconis bin width, (b) the ECDF value at 30 ms, and (c) how FD and Sturges behave for the full n = 200 set.

(a) FD width needs the IQR. With linear-interpolation percentiles on n = 8 sorted points, the Q1 position is 0.25 × (8−1) = 1.75 — three-quarters of the way from the 2nd value (15) to the 3rd (18):

The Q3 position is 0.75 × 7 = 5.25 — a quarter of the way from the 6th value (45) to the 7th (80):

So IQR = 53.75 − 17.25 = 36.5. The cube root of n is 8^1/3 = 2. Therefore:

(b) The ECDF at 30 ms. Count the points at or below 30: {12, 15, 18, 22, 30} — that is 5 of 8.

(c) FD and Sturges for n = 200, IQR = 45. The cube root of 200 is 5.848, so:

Notice how differently they behave. FD fixes the width from the data's spread; Sturges fixes the count from n alone and ignores spread entirely. Concretely: our n = 200 data spans some range R. Sturges says "use 9 bins" no matter what R is, so a dataset spanning 100 ms and one spanning 1000 ms both get 9 bins — the second gets 10x-wider bins and loses all fine structure. FD, keyed to the IQR (45 ms here), says "use 15.4 ms bins" regardless of a lone 400 ms outlier that would wreck any range-based rule. That robustness to outliers is exactly why FD is the safer default on latency and loss data, which are almost always right-skewed — but the only truly safe move is to check the ECDF too.

python — from scratch
import numpy as np
d = np.array([12,15,18,22,30,45,80,210.])

# manual ECDF: sort, then running fraction
xs = np.sort(d)
ys = np.arange(1, len(d)+1) / len(d)     # [1/8, 2/8, ... , 8/8]
# plt.step(xs, ys, where='post')  -> the staircase

# manual FD width
q1, q3 = np.percentile(d, [25, 75])
width = 2 * (q3 - q1) / len(d)**(1/3)
print(q1, q3, q3-q1, width)          # 17.25 53.75 36.5 36.5
print(np.mean(d <= 30))               # 0.625

python — library equivalent
import matplotlib.pyplot as plt
plt.hist(data, bins='fd')          # Freedman-Diaconis, chosen for you
plt.ecdf(data)                     # matplotlib >= 3.8
# or: import seaborn as sns; sns.ecdfplot(data)

We can now see distribution shape honestly. But the moment we start comparing two distributions with summary numbers, we need to put uncertainty on those numbers — and that is where three different whiskers, differing by a factor of five, cause more confusion than any other plotting decision. That is Chapter 2.

Chapter 2: Error Bars Done Right — SD, SE, CI, and the Overlap Fallacy

You added whiskers to your bars — good instinct. But three different whiskers are commonly drawn on identical data, they differ by a factor of four, and most readers cannot tell you which one they are looking at. An unlabeled error bar is not evidence; it is decoration.

Start with the scandal. The same eval — mean 0.72, standard deviation 0.10, over n = 25 fine-tuning runs — can honestly wear three different whiskers. SD whiskers (±0.10) describe the spread of individual runs. SE whiskers (±0.02) describe the uncertainty in the mean. 95% CI whiskers (±0.041 with the small-sample correction) describe a range that would capture the true mean 95% of the time. Same data; whisker widths 0.10 vs 0.02 vs 0.041 — a 5x range. Without a caption saying which is which, the plot is meaningless.

When do you use which? Use SD when the reader should care about run-to-run variability itself — "how reproducible is this RL training?" Use SE or CI when the claim is about the mean — "model B's average score is higher." The CI is SE made interpretable: for n = 25 you multiply SE by the t critical value 2.064, not 1.96, because small samples need Student's t (the why lives in eval-statistics; here we just consume the number).

Walk the arithmetic pipeline once, by hand. SD = 0.10, n = 25, so the standard error is:

The normal 95% half-width would be 1.96 × 0.02 = 0.0392. The t-corrected half-width, with 24 degrees of freedom, is:

The t correction widened the interval by about 5% — small, but honest. Using 1.96 on n = 25 slightly understates your uncertainty every time.

Now the fallacy that fools even reviewers: "the CIs overlap, so the difference is not significant." This is false. Take two means, 0.72 and 0.78, each with SE 0.02. Their 95% CIs are [0.681, 0.759] and [0.741, 0.819] — they overlap by 0.018. Yet the test of the difference uses SE_diff = √(0.02² + 0.02²) = 0.0283, giving z = 0.06 / 0.0283 = 2.12, p = 0.034: significant at the 5% level. Overlapping CIs and a significant difference, both true at once.

Why? The difference's standard error grows by √2, not by 2. Comparing whisker tips implicitly demands the means be 2 × 1.96 = 3.92 SEs apart, but significance only needs 1.96 × √2 = 2.77 SEs apart. The visual "do the whiskers overlap" test is too conservative by a factor of √2 — it will make you dismiss real effects.

The professional fix: when the comparison is the point, plot the difference with its own CI and ask whether it excludes zero. Or use the 84% CI trick — two 84% intervals that just touch correspond to roughly p = 0.05 for two groups with similar SEs (we derive this exactly in Chapter 9). Difference-plots also compose beautifully: a forest plot of per-benchmark differences, each with its own CI, is the honest way to report a model comparison across a dozen benchmarks at once.

Paired data changes everything. If both models are evaluated on the same examples, plot the distribution or CI of the per-example differences. The pairing cancels example difficulty — a hard example that both models fail contributes nothing to the difference — and this often shrinks the interval dramatically. Showing two independent-looking CIs for paired data throws away exactly the information that makes the eval sensitive. (The paired-test math lives in eval-statistics; the plot rule is simply: paired data gets a difference plot.)

Make caption discipline a hard rule. Every error bar in every figure states (1) what statistic — SD, SE, or 95% CI; (2) the n; and (3) what unit is being resampled — runs? examples? seeds? Anthropic's 2024 note "Adding Error Bars to Evals" (Evan Miller) argued that most LLM eval reports omit all three, and made exactly this a norm-setting demand for the field.

The simulation lets you feel the fallacy. On the left, one group's runs are drawn as jittered dots with all three whisker styles on the same mean, each labeled with its live width. On the right, two group means wear 95% CIs, plus a mini-axis showing the difference with its own CI and a verdict lamp. Drag group B's mean up and down; watch for the zone where the CIs overlap but the difference-CI still excludes zero — the fallacy zone glows.

Worked example: three whiskers, then the overlap fallacy

Part 1 — the three widths. 25 fine-tuning runs, mean 0.72, SD 0.10.

Part 2 — overlap yet significant. Two models, means 0.72 and 0.78, each SE 0.02. Using the normal half-width 0.0392:

They overlap: A's top (0.7592) exceeds B's bottom (0.7408) by 0.7592 − 0.7408 = 0.0184. Now test the difference directly:

Significant at 5%, despite the visible overlap. If you had trusted the whiskers, you would have shipped the wrong conclusion.

python — from scratch
import numpy as np
from scipy import stats

se = 0.10 / np.sqrt(25)                        # 0.02
half_n = 1.96 * se                             # 0.0392
tcrit  = stats.t.ppf(0.975, 24)               # 2.0639
half_t = tcrit * se                           # 0.0413

# the overlap fallacy, quantified
overlap = (0.72+half_n) - (0.78-half_n)   # 0.0184
sed = np.sqrt(0.02**2 + 0.02**2)          # 0.02828
z = 0.06 / sed                              # 2.121
p = 2 * (1 - stats.norm.cdf(z))            # 0.034
print(round(overlap,4), round(sed,5), round(z,3), round(p,3))

python — library equivalent (seaborn bootstraps the CI)
import seaborn as sns
sns.pointplot(data=df, x='model', y='score', errorbar=('ci', 95))
# but the honest comparison plots the DIFFERENCE with its own CI:
d, sed = mb - ma, np.sqrt(sea**2 + seb**2)
ax2.errorbar([0], [d], yerr=1.96*sed); ax2.axhline(0)

We now put honest uncertainty on any comparison of summary numbers. But the single plot every ML engineer stares at most is not a bar chart — it is a training curve, and its default dashboard view is quietly wrong on both axes. Chapter 3.

Chapter 3: Training Curves — Smoothing, Log Scales, and Shape Literacy

The single plot every ML engineer stares at most is a loss curve. And the default dashboard view — linear y-axis, heavy smoothing — is quietly wrong on both axes at once. Let us fix both, and then learn to read curve shape like a cardiologist reads an ECG.

Frame it with a code-completion model pretraining run. Loss falls 8.0 → 4.0 in the first 2% of steps, then 4.0 → 2.0 over the entire rest of the run. On a linear y-axis, that second phase — where the model's actual competence is being won — is a nearly flat line hugging the bottom. The plot says "nothing is happening" while perplexity is halving.

The fix is a log-scale y-axis. On a log axis, equal vertical distances represent equal ratios, so 8→4 (halving) and 4→2 (halving) have identical visual size. Loss improvements are multiplicative, so the axis should be multiplicative too. This is not a trick to inflate progress — it is matching the encoding to the quantity, exactly the Chapter 0 principle.

Log-x is often right as well. Learning dynamics spread over orders of magnitude of steps, and the loss-versus-compute curves of scaling-law papers are log-log for exactly this reason (Chapter 5 completes that story). But first, the smoothing problem, which is subtler and more dangerous.

Raw per-step loss is noisy because each step sees a different random batch. Dashboards tame this with an exponential moving average (EMA): smoothed = β × previous + (1 − β) × current. The knob β sets an effective averaging window of about 1 / (1 − β) steps. So β = 0.9 is a 10-step memory, β = 0.99 a 100-step memory, and β = 0.999 — a common dashboard default! — a 1000-step memory.

Here is the catch: smoothing is a low-pass filter, and every filter lags. We will work this exactly below: a loss sequence dropping 4.0 → 2.0 over five readings, smoothed with β = 0.9, reads 2.64 at the moment the true loss is 2.00. The display is 0.64 behind reality. The consequences are real: a loss spike appears later and smaller than it actually was; a divergence looks like a "slight uptick" for hundreds of steps; and comparing two runs at "the same step" compares different effective times if their smoothing differs.

Bias correction. An EMA started at zero is dragged toward zero early in training. Dividing by (1 − β^t) fixes it. Without correction, the first readings of every run look artificially good (lower loss than reality) — so two runs compared in their first hundred steps can swap places purely from an initialization artifact. TensorBoard's smoothing slider applies a debiased EMA; W&B offers several — know which your dashboard uses before comparing screenshots across tools.

Now the shape gallery — read curves like an ECG. (1) Healthy: a fast initial drop, then a smooth power-law-ish decay. (2) LR too high: large oscillations, or the loss plateauing high with sawtooth noise. (3) LR too low: painfully slow, nearly linear descent. (4) Divergence: exponential blow-up, often right after a spike — check gradient norms. (5) Overfitting: training loss keeps falling while validation loss bottoms out and rises — the gap is the signal, which is why you must always plot train and val on the same axes. (6) Data bug or repeated data: sudden cliff drops or periodic scalloping aligned with epoch boundaries.

Let us make the log-y point concrete with numbers. On a linear axis spanning 0 to 8, the drop from 4.0 to 2.0 is a vertical move of 2 units out of 8 — a quarter of the frame — while the earlier drop 8.0 to 4.0 is also 4 units, so the early phase visually dominates and the late phase looks like a flat tail. On a log axis, 8→4 spans log₂(8/4) = 1 unit and 4→2 spans log₂(4/2) = 1 unit: identical. The late competence-building phase gets equal visual weight to the early collapse, because it represents an equal fractional improvement.

Comparison discipline for curves: align runs on a meaningful x-axis (tokens or FLOPs, not wall-clock — wall-clock differences are hardware, not learning), use the same smoothing for all runs, and never crop the y-range to make one run's advantage fill the frame without annotating the range (the Chapter 0 rule: lines may zoom, but must say so).

One more consequence of lag worth internalizing: it is asymmetric in how it deceives. During improvement, the smoothed curve reads worse than reality (as in our worked example, 2.64 vs 2.00). During a spike, the smoothed curve reads better than reality — a loss that momentarily jumps to 6.0 appears on a β = 0.99 dashboard as a gentle bump to maybe 4.1, arriving dozens of steps late. That is the dangerous direction: the one place you most need to see a problem immediately (a divergence) is exactly where heavy smoothing hides it longest.

The eval connection closes the chapter. Intermediate checkpoint evals are points on these curves; a benchmark score plotted every N steps has its own sampling noise (Chapter 2's error bars), so a "dip at checkpoint 40k" that sits inside the CI band is not a regression. Curve literacy plus uncertainty literacy prevents the classic incident: "we reverted a fine run because one noisy checkpoint eval dipped."

The viewer below plots a synthetic 2000-step run with raw loss (faint), the EMA (bold), and optional validation. Inject a pathology, drag the β slider, and watch the lag readout — the gap between what the dashboard shows and the true underlying loss. Toggle log-y and bias correction; turn correction off and watch the first 50 steps sag artificially.

Worked example: the lag of an EMA, step by step

Five consecutive loss readings: 4.0, 3.0, 2.5, 2.2, 2.0. Smooth with β = 0.9, initialized at 0, with bias correction. The update is e_t = 0.9 e_t−1 + 0.1 L_t, and the corrected value is e_t / (1 − 0.9^t).

The dashboard displays 2.64 while the true current loss is 2.00 — a lag of 0.64. That is roughly the 1/(1−β) = 10-reading memory showing through: during rapid improvement the smoothed value is systematically above the truth, because it still remembers the higher losses from up to ten readings ago.

python — from scratch
def ema(xs, beta):
    out, e = [], 0.0
    for t, x in enumerate(xs, 1):
        e = beta*e + (1-beta)*x
        out.append(e / (1 - beta**t))    # bias-corrected
    return out

print([round(v,4) for v in ema([4.0,3.0,2.5,2.2,2.0], 0.9)])
# [4.0, 3.4737, 3.1144, 2.8485, 2.6413]   -> reads 2.64 while truth is 2.00
print(1/(1-0.9))   # 10.0  effective window

python — library equivalent (pandas)
import pandas as pd
smoothed = pd.Series(loss).ewm(alpha=1-beta, adjust=True).mean()
# adjust=True IS the bias correction; plot raw at alpha=0.25, smoothed bold, ax.set_yscale('log')

We can now read one model's learning honestly. But comparing two classifiers means decompressing an entire trade-off surface — and each of the four workhorse comparison plots is routinely read wrong. Chapter 4.

Chapter 4: Comparison Plots — ROC, PR, Calibration, and Residuals

One accuracy number compresses an entire trade-off surface into a single dot. Four plots decompress it — ROC, precision-recall, reliability diagrams, and residual plots — but each one answers exactly ONE question and each one is routinely read wrong.

A scope note first. The definitions of these metrics — what AUC, ECE, and precision mean and why — are metric-design's territory (it derives the AUC-as-ranking-probability identity and hand-computes ECE). This chapter owns reading and plotting them honestly: axes, operating-point dots, curve crossings, error bands, and picking the plot that answers the decision at hand. Running example: a camera-trap classifier flagging a rare species. A score threshold sweeps from strict to lenient; every threshold is a different operating point.

The ROC curve plots the true-positive rate (TPR) against the false-positive rate (FPR) across all thresholds. It shows the model's ranking quality and is immune to class prevalence, because both axes are rates within their own class. Read it correctly: the diagonal is coin-flipping; the closer the curve hugs the top-left corner the better; and AUC is the probability that a random positive outranks a random negative (that identity is proven in metric-design; here we just use it to read the curve).

But prevalence-immunity is a double-edged sword. With 3 rare positives per 97 negatives, an innocuous-looking FPR of 0.14 means 0.14 × 97 ≈ 14 false alarms drowning out 2 true detections. The ROC looks great; the product is unusable. ROC's blindness to rarity is precisely how imbalanced-data papers oversell.

The PR curve plots precision against recall and puts prevalence back in, because precision = TP / (TP + FP) directly feels every false alarm. In the worked example below, the same operating point (TPR 0.667, FPR 0.143) has precision 0.667 on a balanced set but only 0.126 when negatives outnumber positives 97:3. The rule: rare-positive problems — fraud, rare species, safety violations — get PR curves. ROC alone hides the collapse.

Neither curve is a single number, and models can cross: A better at low FPR, B better at high recall. So plot the curves, mark your operating point — the threshold you actually deploy — with a dot, and compare at that dot. AUC differences of 0.005 are usually within resampling noise anyway (error bands via bootstrap, machinery in eval-statistics).

Calibration / reliability diagrams answer a different question: when the model says it is 80% confident, is it right 80% of the time? Bin predictions by confidence, then plot mean confidence vs empirical accuracy per bin; the identity diagonal is perfect calibration. Points below the diagonal mean overconfidence — the modern deep-net default. Summarize the gap with ECE (expected calibration error), the prediction-weighted average gap between confidence and accuracy. Metric-design owns the ECE definition and its Brier-score cousin; here we read the diagram and note its fragility — ECE inherits the histogram's bin-count sensitivity from Chapter 1.

Why calibration plots matter for eval work: LLM-as-judge scores, uncertainty-aware routing, abstention thresholds, and any downstream system consuming probabilities all silently assume calibration. A model that "improves accuracy" while becoming wildly overconfident can make the system worse — and only the reliability diagram reveals this.

For regressors and quality scores, the workhorse is the predicted-vs-true scatter with the identity line y = x. Points on the line are perfect; the vertical gap to the line IS the error; systematic curvature away from it is bias; and points hugging a horizontal band mean the model predicts the mean and ignores its input. Never fit-and-plot a regression line instead of the identity line — the regression line always looks flattering because it is optimized to.

Residual plots (residual = predicted minus true, plotted against the prediction or an input) reveal error structure. A shapeless cloud around zero is health. A fan opening rightward is heteroscedasticity — errors growing with magnitude, like a depth estimator whose error grows with distance. A curve means a missing nonlinearity. Stripes or clusters mean a data slice with distinct behavior — which is Chapter 6's cue to go slice-level.

Read residual structure like a diagnosis, not a verdict. Suppose a depth estimator's residuals form a fan that opens rightward: near residuals cluster within ±0.1 m, but at 20 m predictions the residuals spread ±2 m. That is heteroscedasticity, and it has a concrete engineering consequence — a single RMSE number averages the tight near-field and loose far-field into one misleading figure, so you should report error as a function of distance, not one scalar. The residual plot told you which model of the noise to build; the scalar metric would have hidden it entirely.

The meta-skill: pick the plot that answers the decision. Choosing a deploy threshold → PR curve with your operating point. Trusting probabilities → reliability diagram. Debugging a regressor → residuals. Publishing ranking quality → ROC with error bands. A plot chosen because it looks impressive rather than because it answers the decision is Chapter 0's lie in a fancier costume.

The simulation links three panels driven by one synthetic classifier. Drag the threshold and watch the operating point move on both curves. The prevalence slider is the lesson: as positives become rare, the ROC curve does not move while the PR curve collapses. Flip to the calibration view and use the temperature slider to make the model over- or under-confident, watching ECE respond.

Worked example: the prevalence collapse and a 2-bin ECE

At one threshold the camera-trap classifier gives TP = 2, FN = 1, FP = 1, TN = 6 (3 positives, 7 negatives). Compute the rates:

Now keep the same per-class rates but change the population to 3 positives and 97 negatives. Expected TP = TPR × 3 = 2 (unchanged). Expected FP = FPR × 97 = 97/7 = 13.857. So:

Same ROC point (0.1429, 0.6667); precision fell from 0.667 to 0.126 — an 81% relative collapse, purely from prevalence. The ROC could not show it; the PR curve does.

ECE from two bins. Bin 1 has 10 predictions, average confidence 0.90, accuracy 0.70. Bin 2 has 10 predictions, average confidence 0.60, accuracy 0.65. The per-bin gaps are |0.90 − 0.70| = 0.20 and |0.60 − 0.65| = 0.05. The prediction-weighted average:

The dominant bin-1 gap is above-the-diagonal (confidence 0.90 > accuracy 0.70), so the model is overconfident overall.

python — from scratch
TP, FN, FP, TN = 2, 1, 1, 6
tpr, fpr = TP/(TP+FN), FP/(FP+TN)
fp_imb = fpr * 97                     # expected FP under 3:97
print(round(tpr,4), round(fpr,4),
      round(TP/(TP+FP),4), round(2/(2+fp_imb),4))
# 0.6667 0.1429 0.6667 0.1261

# ECE from two bins: prediction-weighted |conf - acc|
ece = (10*abs(0.9-0.7) + 10*abs(0.6-0.65)) / 20
print(ece)                            # 0.125

python — library equivalent (sklearn)
from sklearn.metrics import RocCurveDisplay, PrecisionRecallDisplay
RocCurveDisplay.from_predictions(y, scores)
PrecisionRecallDisplay.from_predictions(y, scores)
from sklearn.calibration import calibration_curve   # reliability diagram bins

We can now decompress any classifier comparison. But the plots that drive the biggest decisions — which component to keep, how big to build — are ablations and scaling sweeps, and they attract the most sophisticated lies of all. Chapter 5.

Chapter 5: Ablation and Scaling Plots — Log-Log Honesty

Ablations and scaling sweeps drive the biggest decisions in ML — which component to keep, how large to build the next model — and precisely because the stakes are highest, they attract the most sophisticated lies. This chapter arms you against them, and teaches you to fit a power law by hand.

Ablation tables become ablation plots when you need to see interaction and uncertainty at a glance: a grouped bar chart where groups are benchmarks or slices and the bars within a group are variants (full model, minus component X, minus Y), each bar wearing a 95% CI whisker from Chapter 2. Zero baseline (they are bars!), consistent variant colors across all groups, and the full-model bar always in the same position so the eye can sweep.

The ablation-plot lie catalog: sorting groups by the favored variant's advantage (cherry-ordering); dropping the benchmarks where the ablation actually helped (survivorship); whiskerless bars at n = 1 seed; and the Chapter 0 truncated baseline. An honest ablation plot shows ALL evaluated benchmarks, ordered alphabetically or by task family, with the per-bar n stated.

When one factor is dose-like — data fraction, model width, compute — lines beat bars: x = dose, y = metric, one line per variant, a CI band per line. Now curvature becomes visible. "Component X helps at small scale but washes out at large scale" is a crossing of two lines — and it is invisible in any table.

Now scaling laws. Loss versus scale follows an approximate power law, L = a × N^−α. On linear axes this is a boring hyperbola. On log-log axes a power law is a perfectly straight line whose slope IS the exponent, because taking logs of both sides gives log L = log a − α log N. That is why every scaling-law figure since Kaplan 2020 is log-log: straightness is the hypothesis test your eye performs.

Work the fit by hand. Three training runs of the code-completion model at N = 1e6, 1e7, 1e8 parameters give losses 4.00, 3.17, 2.52. In log₁₀ space the x-values are (6, 7, 8) and the y-values are (0.6021, 0.5011, 0.4014). The least-squares slope comes out to −0.1003, so α = 0.100: every 10x in parameters cuts loss by the factor 10^0.1003 = 1.26, i.e. about 21% off. The intercept 1.2038 completes the line. (Full arithmetic in the worked example.)

Extrapolation is where scaling plots turn into fiction. Our fitted line predicts L(1e9) = 2.00 — one full order of magnitude beyond the data. The honest rendering shades the region beyond the last data point, switches the line to dashed, and widens the uncertainty band with distance. Three points spanning two decades cannot certify decade three: power laws bend when data, resolution, or task ceilings bite. The field's own history proves it — Chinchilla revised Kaplan's compute-optimal recipe in 2022, then Epoch AI's 2024 replication revised Chinchilla's own fitted constants. Both were disputes about fits and their extrapolations.

Log-log reading traps. (1) Equal visual gaps are equal ratios — a hair of daylight between two curves at the right edge can be 5% of loss, which at that scale is enormous. (2) Zero cannot appear on a log axis, so metrics that hit zero (error rates on easy tasks) get quietly clipped, or the plot silently switches to log-linear — check which. (3) Straightening by choice of variable: some papers plot against compute, some against parameters, some against data, and a curve straight in one variable can bend in another. Choosing the straightest x is a subtle form of result-shopping.

The emergence controversy is this chapter's capstone in plot literacy. "Emergent abilities" — flat-then-cliff curves of accuracy vs scale — were shown by Schaeffer et al. (NeurIPS 2023 best paper) to largely be artifacts of the metric and the plot: nonlinear, discontinuous metrics like exact-match manufacture cliffs out of smoothly improving log-likelihood. Same models, different y-variable, opposite narrative. The deepest lesson of the chapter: the choice of what to put on the axes is itself a modeling decision.

The fitting bench below plots loss vs parameter count. Toggle log-log and watch the hyperbola straighten. The least-squares line renders with its live slope; beyond the last point it turns dashed inside a shaded zone. Drag a data point to see fit sensitivity, add a 4th off-trend run to watch the extrapolation shift, and drag the target-scale marker out to 1e10 — the caveat badge grows with distance.

Worked example: fitting a power law in log space

Three runs: N = 1e6 → loss 4.00; N = 1e7 → 3.17; N = 1e8 → 2.52. Fit L = a × N^−α by least squares in log₁₀ space, then extrapolate to 1e9.

Transform. x = log₁₀N = (6, 7, 8); y = log₁₀L = (0.60206, 0.50106, 0.40140). The means:

Deviations. dx = (−1, 0, 1); dy = (0.10055, −0.00045, −0.10011). The least-squares slope is Σ(dx dy) / Σ(dx²):

So α = 0.1003. The intercept is ȳ − slope × x̄:

Extrapolate to x = 9:

Interpretation: each 10x of parameters multiplies loss by 10^−0.10033 = 0.794 — a 21% reduction per decade. And the 1e9 prediction sits one full decade beyond the data, so it must be drawn dashed inside a shaded zone. It is a hypothesis, not a measurement.

python — from scratch
import numpy as np
x = np.log10([1e6, 1e7, 1e8])          # [6, 7, 8]
y = np.log10([4.00, 3.17, 2.52])

dx, dy = x - x.mean(), y - y.mean()
slope = (dx*dy).sum() / (dx**2).sum()   # -0.10033
intercept = y.mean() - slope * x.mean()   # 1.20382
L9 = 10**(intercept + slope*9)               # ~2.00
print(round(slope,5), round(intercept,5), round(L9,3), round(10**slope,4))
# -0.10033 1.20381 1.999 0.7937

python — library equivalent
slope, intercept = np.polyfit(np.log10(N), np.log10(L), 1)
ax.set_xscale('log'); ax.set_yscale('log')
ax.axvspan(N[-1], N_target, alpha=0.1)   # the honesty shade

We can now keep the highest-stakes plots honest. But there is a subtler failure that no amount of axis discipline catches: an aggregate metric that rises while a whole subgroup collapses beneath it. Chapter 6.

Chapter 6: Slice Heatmaps and Metrics Over Time

Your overall accuracy went up with the new model. You shipped. Two weeks later, support tickets from one user segment tripled. The aggregate plot never had a chance of warning you — and this chapter's plots are the ones that would have.

Start with the arithmetic of hiding. Overall accuracy is a size-weighted average of slice accuracies. Camera-trap example: daytime images (n = 900, accuracy 0.92) and nighttime images (n = 100, accuracy 0.50) give overall (900×0.92 + 100×0.50)/1000 = 0.878. A new model scores daytime 0.95 but nighttime 0.40: overall (855 + 40)/1000 = 0.895. The headline metric improved by 1.7 points while nighttime performance collapsed by 10. Nothing about the aggregate is wrong; it is answering a different question than "is anyone worse off?"

The slice heatmap is the antidote: rows are slices (device, language, image condition, input length, user segment), columns are models or versions, cell color encodes the metric, and cell text prints the value AND the n. Color encodes where to look; the printed n encodes whether to believe it. A 0.40 cell with n = 100 carries a CI of roughly ±0.10 and deserves attention; a 0.63 cell with n = 8 deserves a bigger sample before anyone panics.

Heatmap craft rules. (1) Use diverging colormaps only when there is a meaningful midpoint (like delta-vs-baseline centered at 0); use sequential colormaps for raw levels — a red-to-green map of raw accuracy invents a "neutral" point nobody chose. (2) Normalize per row when slices have wildly different baseline difficulty, else one hard slice's dark row hides all within-row structure. (3) Sort rows by delta or by n; never leave dictionary order if the plot's job is triage. (4) Print values in cells whenever the grid is smaller than about 15×10 — color perception is not precise enough for decisions.

Better than two heatmaps (old model, new model) side by side is ONE delta heatmap: each cell is new minus old, a diverging colormap centered at zero, and cells are hatched out when the change is within the slice's noise band. This single plot is the regression-review workhorse — regression-testing-ml builds its gate logic on exactly this view; here we own making it readable.

Metrics over time. The deployed model's metric plotted per day or week is a time series, and honest ones carry two garnishes. (1) An uncertainty band — with about 1000 eval examples per week at p = 0.878, the weekly standard error is about 0.0104, so a ±2 SE band of ±0.021 tells the on-call reader which wiggles are noise. (2) Event annotations — vertical lines for every deploy, data-pipeline change, and eval-set refresh. An unannotated step change is a mystery; an annotated one is a diagnosis.

Drift panels. When the metric moves, the first question is "did the model change or did the inputs change?" A drift panel pairs the metric time series with input-distribution summaries over time (feature means, class mix, input-length ECDF snapshots) as small aligned subplots sharing the x-axis. Shared-x alignment is the craft point — the reader's eye does the correlation between an input shift and a metric move.

Sparklines — word-sized metric-over-time strips embedded one per table row — combine both views: a table of slices where each row shows current value, delta, n, and a tiny 8-week trend line. This is the densest honest format for a weekly eval review. It fails only when axes differ silently per row, so fix a shared y-range per metric column and say so.

Read the heatmap's n column as a believability filter, with a number attached. A slice cell showing accuracy 0.40 on n = 100 examples has a standard error of √(0.40 × 0.60 / 100) = √0.0024 = 0.049, so its rough 95% CI is ±0.096 — nearly ±0.10, wide enough that a one-week drop from 0.50 to 0.40 might partly be noise and warrants a bigger sample before a rollback. A cell at n = 8 has SE around 0.17: essentially unreadable. Color tells you where to look; the printed n and its implied CI tell you whether to act.

A boundary note. These are all eval surfaces — is the model good, where, and since when. Once the y-axis becomes revenue, retention, or tickets, you have crossed into product dashboards, and the metrics-ladder lesson owns that layer, including how eval metrics should (and should not) be composed into exec-facing KPIs.

The split view below shows the aggregate accuracy as one big number and trend line on top, and a slice heatmap (daytime / nighttime / rain / snow × weekly versions) below. Ship a new version and occasionally watch the aggregate tick up and glow green while a minority-slice cell flashes red. The delta toggle switches the heatmap to new-minus-old with noise cells hatched; the drift toggle adds an input-mix panel sharing the x-axis.

Worked example: the average rises, the slice collapses

Old model: daytime slice n = 900 at 0.92, nighttime n = 100 at 0.50. New model: daytime 0.95, nighttime 0.40.

The aggregate change is +0.017 (up 1.7 points). The nighttime change is 0.40 − 0.50 = −0.10 (down 10 points, a 20% relative collapse). It is hidden because the slice holds only 10% of the weight: its −0.10 contributes just −0.01 to the aggregate, outweighed by daytime's +0.03 × 0.9 = +0.027.

The weekly band. For an overall accuracy of 0.878 measured on 1000 examples:

A ±2 SE band is ±0.0207, so weekly readings between 0.857 and 0.899 are indistinguishable from a flat 0.878. Any wiggle inside that band should never trigger a discussion — drawing the band is what stops false alarms.

python — from scratch
old = (900*0.92 + 100*0.50) / 1000     # 0.878
new = (900*0.95 + 100*0.40) / 1000     # 0.895
import numpy as np
se = np.sqrt(0.878 * 0.122 / 1000)          # 0.01035
print(old, new, round(se, 5))                 # 0.878 0.895 0.01035

python — library equivalent (seaborn heatmap)
import seaborn as sns
sns.heatmap(delta_df, annot=True, fmt='.2f', center=0, cmap='RdBu_r')
# center=0 is the honesty flag: the colormap diverges around no-change

You now hold every lie individually. The showcase lets you stack them — and watch a true conclusion transform, toggle by toggle, into a false one. Chapter 7.

Chapter 7: Showcase — The Honest Plot Builder

You now know each lie individually. The showcase lets you stack them — and watch a true conclusion ("no significant difference") transform, toggle by toggle, into a false one ("B is a breakthrough"). Every distortion from Chapters 0 through 6 composes into one machine.

Meet the fixed ground truth, baked into the simulation. Two code-completion models, A and B, each trained with 5 seeds and evaluated on the same benchmark. The true story: mean A = 0.8490, mean B = 0.8502, with per-seed scatter much larger than the 0.12-point gap. Welch's t = 0.45, p = 0.67. Any rendering that leads a reader to "B is clearly better" is, by construction, a lie about this data.

Five levers, each mapped to its chapter. (1) Axis truncation (Ch0) — the baseline slides from 0 up to 0.84. (2) Error bars (Ch2) — off / SD / SE / 95% CI. (3) Seed selection (this chapter's new lever) — "mean of 5 seeds" vs "best seed per model" vs the maximally dishonest "best B seed vs mean A." (4) Smoothing (Ch3), applied to the per-checkpoint curve view. (5) Log scale (Ch5) on the curve view's y-axis, used here to compress B's late-training noise.

Let us quantify lever 3, because it is the one this chapter owns. Cherry-picking B's best seed (0.856) against A's mean (0.849) reports a 0.7-point gap where the true mean gap is 0.12 points — a 5.8x inflation, before any axis games at all. With 5 seeds, the expected maximum sits about one standard deviation above the mean, so single-best-seed reporting builds in roughly a +1 SD bias for free. Papers reporting best-of-N runs without stating N commit this silently.

The live conclusion caption is the pedagogical heart. A rule-based reader looks at the current rendering the way a hurried reviewer would — visual gap versus whisker width versus baseline — and prints what that reviewer would conclude. Honest settings print "A and B are statistically indistinguishable (p = 0.67)." Full-lie settings print "B delivers a dramatic improvement." A running LIE SCORE from 0 to 5 counts how many levers are currently distorting.

The reverse game teaches detection. A "detective mode" button renders the plot with hidden lever settings, and you must name every active lie before revealing. This is the skill transfer that matters: from building honest plots to auditing hostile ones, which is exactly what interviews and code reviews demand.

Connect it to the real world. The 2025 "Leaderboard Illusion" analysis (Singh et al., arXiv April 2025) of Chatbot Arena documented private best-of-N variant testing with selective disclosure of only the best performer — lever 3 operating at industrial scale. Multi-seed reporting norms (mean ± CI over seeds, all seeds shown as dots) exist precisely to make this lever visible.

Codify the exit checklist — the Honest Plot Contract, five clauses: (1) zero-baseline or annotated zoom; (2) uncertainty shown and labeled (statistic + n + unit); (3) all runs / seeds / slices disclosed, with selection rules stated; (4) axes and smoothing declared; (5) the conclusion stated WITH its uncertainty. Every figure in your next eval report should pass all five.

Below is the full builder. Adjust all five levers, switch between bar view and curve view, and watch the conclusion caption rewrite itself and the LIE SCORE climb. Every seed is drawn as a dot beside its bar, so seed-selection visibly grays out excluded dots. Hit "detective mode" to hide the levers and quiz yourself; "reset to honest" returns to a lie score of 0.

Worked example: the honest comparison, and the cherry-pick inflation

Seed scores. Model A: 0.850, 0.847, 0.852, 0.845, 0.851. Model B: 0.849, 0.856, 0.851, 0.842, 0.853.

SD of A. Deviations from 0.8490: (0.001, −0.002, 0.003, −0.004, 0.002). Squares sum = (1 + 4 + 9 + 16 + 4)×10⁻⁶ = 34×10⁻⁶. Variance = 34e−6 / 4 = 8.5e−6, so SD_A = 0.00292.

SD of B. Deviations from 0.8502: (−0.0012, 0.0058, 0.0008, −0.0082, 0.0028). Squares sum = (1.44 + 33.64 + 0.64 + 67.24 + 7.84)×10⁻⁶ = 110.8e−6. Variance = 27.7e−6, so SD_B = 0.00526.

Welch's test.

Honest conclusion: indistinguishable. Now the cherry-pick. Best B seed = 0.856; reported gap vs mean A = 0.856 − 0.849 = 0.0070.

The reported 0.7-point gap is 2.6 times B's own seed SD — it advertises the luckiest draw as the typical result. Axis honesty cannot repair this; the lie lives in the data selection, upstream of any rendering.

python — from scratch (verified with scipy)
import numpy as np
from scipy import stats
A = np.array([0.850,0.847,0.852,0.845,0.851])
B = np.array([0.849,0.856,0.851,0.842,0.853])
t, p = stats.ttest_ind(B, A, equal_var=False)
print(A.mean(), round(B.mean(),4),
      round(A.std(ddof=1),5), round(B.std(ddof=1),5),
      round(t,3), round(p,2))
# 0.849 0.8502 0.00292 0.00526 0.446 0.67

gap_true, gap_cherry = B.mean()-A.mean(), B.max()-A.mean()
print(round(gap_true,4), round(gap_cherry,4), round(gap_cherry/gap_true,2))
# 0.0012 0.007 5.83

python — library equivalent (dots + honest interval, two lines)
import seaborn as sns
sns.stripplot(data=long_df, x='model', y='score', color='gray')
sns.pointplot(data=long_df, x='model', y='score', errorbar=('ci',95))

The showcase turned every lesson into one interactive audit. Chapter 8 turns it into reusable code — the plots-as-code module that makes honesty a library default rather than an act of memory.

Chapter 8: The Recipe Box — matplotlib and pandas for Every Plot

Knowledge without recipes evaporates at the terminal. This chapter is the cookbook: every plot from Chapters 0 through 7, as code you can paste, in both from-scratch matplotlib and the seaborn one-liner. The organizing idea is one convention that makes plots composable into review dashboards.

The foundational pattern is fig, ax = plt.subplots(). The object-oriented API beats the plt.* state machine for eval work: you get multiple axes, reusable styling functions that take an ax argument, and no global-state surprises when a helper function plots midway through yours. Every recipe below takes ax as its first parameter — that one convention makes plots composable.

Recipe group 1 — distributions (Ch1). The four-line ECDF (sort, arange/n, step plot, grid); the histogram with bins='fd'; and overlay discipline (alpha, zorder, and shared bins via np.histogram_bin_edges on the pooled data so two models' histograms are actually comparable rather than each auto-binned differently).

python — ax-first recipes, group 1
def ecdf(ax, x, **kw):
    xs = np.sort(x)
    ys = np.arange(1, len(x)+1) / len(x)
    ax.step(xs, ys, where='post', **kw)
    ax.grid(True, alpha=0.3)

# comparable histograms: shared edges from the POOLED data
edges = np.histogram_bin_edges(np.concatenate([a, b]), bins='fd')
ax.hist(a, bins=edges, alpha=0.5); ax.hist(b, bins=edges, alpha=0.5)

Recipe group 2 — uncertainty (Ch2). errorbar with capsize for mean+CI dots; fill_between for bands; the difference-plot recipe (a second axis with axhline(0)); and the scatter-of-seeds + pointplot combo so the raw runs are never hidden.

python — ax-first recipes, group 2
def ci_bars(ax, means, ses, labels, z=1.96):
    x = np.arange(len(means))
    ax.errorbar(x, means, yerr=z*np.array(ses), fmt='o', capsize=4)
    ax.set_xticks(x); ax.set_xticklabels(labels)
    return f'mean +/- 95% CI, n stated per point'   # caption baked in

def diff_panel(ax, ma, mb, sea, seb):
    d, sed = mb - ma, np.sqrt(sea**2 + seb**2)
    ax.errorbar([0], [d], yerr=1.96*sed, fmt='s', capsize=5)
    ax.axhline(0, ls='--')   # does the difference clear zero?

Recipe group 3 — curves (Ch3) and trade-offs (Ch4). Raw-plus-smoothed loss with pd.Series.ewm(adjust=True); set_yscale('log'); ROC/PR from-scratch threshold sweeps and their sklearn Display one-liners; the reliability diagram with an identity diagonal via ax.plot([0,1],[0,1]) and equal aspect.

python — ax-first recipes, group 3
def loss_curve(ax, raw, beta=0.9):
    ax.plot(raw, alpha=0.25)                          # raw ghosted
    sm = pd.Series(raw).ewm(alpha=1-beta, adjust=True).mean()
    ax.plot(sm, lw=2); ax.set_yscale('log')

def roc_pr(ax1, ax2, y, s):
    for thr in np.sort(s)[::-1]:
        pred = s >= thr
        tp = ((pred) & (y==1)).sum(); fp = ((pred) & (y==0)).sum()
        # append (fpr, tpr) and (recall, precision), then ax1.plot / ax2.plot

Recipe group 4 — scaling and slices (Ch5, Ch6). loglog with np.polyfit in log space, dashed extrapolation and axvspan shading; imshow + ax.text for heatmap cell annotation, and the seaborn heatmap(annot=True, center=0) equivalent; time series with event axvlines and a fill_between noise band.

python — ax-first recipes, group 4
def loglog_fit(ax, N, L, extrapolate_to):
    x, yv = np.log10(N), np.log10(L)
    slope, b = np.polyfit(x, yv, 1)
    ax.loglog(N, L, 'o')
    xe = np.array([N[-1], extrapolate_to])
    ax.plot(xe, 10**(b + slope*np.log10(xe)), ls='--')   # dashed
    ax.axvspan(N[-1], extrapolate_to, alpha=0.1)         # shade

def delta_heatmap(ax, old, new, n):
    d = new - old; vmax = np.abs(d).max()
    ax.imshow(d, cmap='RdBu_r', vmin=-vmax, vmax=vmax)   # symmetric!
    for i in range(d.shape[0]):
        for j in range(d.shape[1]):
            ax.text(j, i, f'{d[i,j]:.2f}\n n={n[i,j]}', ha='center')

Small multiples — the honest answer to "too many lines on one plot." fig, axes = plt.subplots(3, 4, sharex=True, sharey=True) gives 12 slice panels with comparable axes by construction; loop axes.flat, one slice each, one shared legend. The sharey=True is the honesty feature: per-panel auto-scaling would re-introduce Chapter 0's zoom lie one panel at a time. We work the layout arithmetic below.

Defaults that lie, and their fixes. matplotlib bar charts DO start at zero (good), but plt.plot auto-scales tightly (fine for lines — annotate the range). imshow with the wrong vmin/vmax silently saturates cells, so always set them explicitly for delta maps (symmetric around 0). Legend order defaults to plot order, not importance. And rcParams['figure.dpi'] affects only display, while savefig(dpi=) affects the artifact you ship.

Reproducibility is a plotting concern. A plotting script that takes the eval CSV path and writes the figure is an artifact of record; a notebook cell modified by hand is not. End with the save discipline — fig.savefig('fig_ablation.png', dpi=150, bbox_inches='tight') — plus writing the exact data slice next to the figure so any reviewer can regenerate it. (regression-testing-ml turns this into CI; here we just form the habit.)

The gallery below renders six live recipes from one shared synthetic dataset. Tap a mini-plot to expand it with the exact recipe alongside and 2–3 knobs that live-update it, so you connect each argument to its visual effect. Shuffle the data and watch all six re-render from the same source.

Worked example: laying out honest small multiples

Lay out a 12-slice figure at 4 columns, 2.2 inches per panel, saved at dpi = 150. Also compute honest shared y-limits for slice accuracies ranging 0.61 to 0.94 with 5% padding.

Shared y-limits: the data range is 0.94 − 0.61 = 0.33; the pad is 0.05 × 0.33 = 0.0165; so the limits are:

Because the panels share these limits, a slice whose line sits visibly lower IS lower — no per-panel rescaling can fake parity. Note the deliberate choice: these are line panels, so the 0.59 baseline is legitimate (position encoding), but it gets annotated per the Chapter 0 rule. If the panels were bars, the limits would need to be (0, 1).

python — from scratch
import math
rows = math.ceil(12 / 4)                            # 3
print(rows, 4*2.2, rows*2.2, 4*2.2*150, rows*2.2*150)
# 3 8.8 6.6 1320 990

lo, hi = 0.61, 0.94
pad = 0.05 * (hi - lo)
print(round(lo-pad,4), round(hi+pad,4), round(pad,4))
# 0.5935 0.9565 0.0165

python — the composing figure (library)
fig, axes = plt.subplots(3, 4, sharex=True, sharey=True)
for ax, slice_ in zip(axes.flat, slices):
    timeseries_band(ax, slice_.t, slice_.p, slice_.n, events)
fig.savefig('weekly_review.png', dpi=150, bbox_inches='tight')

You can now build every honest plot in this lesson from a paste-able recipe. Chapter 9 drills the whole lesson as rapid-fire interview Q&A — the skill of auditing a hostile plot in seconds.

Chapter 9: Interview Arsenal — Plot Literacy Under Fire

Every question in this arsenal has appeared, in some costume, in a real ML system-design or analytics interview. Here is the plot: tell me what is wrong, what is missing, and what you would draw instead. Staff-level interviews test whether you can audit a plot in seconds, quantify its distortion, and design the honest replacement.

How plot questions are actually asked: (1) the audit ("here is a figure from a report — critique it"), (2) the design ("what would you plot to answer X?"), (3) the trap ("the CIs overlap, so no difference, right?"), and (4) the judgment call ("leadership wants one number and one chart — what do you give them?"). Each maps to chapters of this lesson.

The 10-second audit routine to verbalize out loud: axes first (baseline? scale? units?), uncertainty second (whiskers? which statistic? what n?), selection third (which runs/seeds/slices made it in?), encoding fourth (does bar-length / color / position match the data type?), conclusion last (does the caption's claim survive the first four checks?).

Quantification is what separates senior from staff answers: not "the axis looks truncated" but "this truncation yields a lie factor of roughly 100"; not "overlapping CIs doesn't mean no difference" but "the whisker-tip test demands 3.92 SEs when significance needs 2.77 — and just-touching 84% CIs are the calibrated visual test" (worked below).

Work the arsenal below — each answer expands. Then the three debate prompts have no single right answer; they exist because real plot decisions are trade-offs, and interviewers push until you take a defensible position.

The Q&A bank

Rapid-fire critique deck

Practice the 10-second audit against randomly generated flawed plots. The deck deals a plot with hidden flaws and a countdown; tap the flaws you spot, then reveal the honest re-render side by side. Run it ten times and narrate your audit aloud each time within 30 seconds.

Worked example: the 84% CI trick, derived

Why do just-touching 84% confidence intervals correspond to roughly p = 0.05 for two independent groups with similar SEs? An 84% CI leaves 16% outside, 8% in each tail, so its z multiplier is the 92nd-percentile z:

Two 84% intervals just touching means the means are separated by exactly 2 × 1.405 × SE = 2.810 SE (each interval contributes 1.405 SE toward the other). The significance test of the difference uses SE_diff = √2 × SE = 1.414 × SE, so the implied test statistic is:

Almost exactly the 1.96 threshold. Compare the naive 95%-whisker-tip test: separation 2 × 1.96 = 3.92 SE implies z = 3.92 / 1.414 = 2.77, p = 0.006 — demanding significance at the 0.6% level while claiming to test at 5%. The 84% rule is the visual test that actually calibrates to p = 0.05.

python — verified with scipy
import numpy as np
from scipy import stats
z84 = stats.norm.ppf(0.92)               # 1.405
z_impl = 2*z84 / np.sqrt(2)             # 1.987
print(round(z84,3), round(z_impl,3),
      round(2*(1-stats.norm.cdf(z_impl)),3))   # 1.405 1.987 0.047

z_naive = 2*1.96 / np.sqrt(2)          # 2.772
print(round(z_naive,3), round(2*(1-stats.norm.cdf(z_naive)),4))  # 2.772 0.0056

Debate prompts (no single right answer)

Chapter 10: Beyond — Where Plot Literacy Plugs In

You can now render — and detect — the truth. Plot literacy is the presentation layer of the whole evaluation stack: statistics feed the intervals, metric design feeds the axes, and every sibling lesson consumes these plots. Here is how it all connects, and where its limits are.

The arc, one sentence per chapter: encodings can lie 121x (Ch0); distributions before summaries, ECDF as the un-liable view (Ch1); label your whiskers and test differences directly (Ch2); smoothing buys clarity with lag (Ch3); pick the trade-off plot that answers the decision (Ch4); power laws are straight only in log-log and only within the data (Ch5); aggregates hide slices (Ch6); lies compose and so must audits (Ch7); honesty belongs in the plotting library, not the plotter's memory (Ch8).

Honest limits of plot literacy: a perfectly honest plot of a badly designed metric is still misleading (metric-design owns what SHOULD be on the y-axis); a beautiful CI is only as good as the statistical assumptions behind it (eval-statistics owns those); and an honest offline plot says nothing about causal product impact (experiment-design owns A/B machinery, metrics-ladder owns the exec translation).

Where this lesson connects

The plot-selection cheat sheet

The formula cheat sheet

The Honest Plot Contract

Worked example: a farewell audit in under a minute

A news-style chart shows quarterly model quality scores 58 and 61 as bars drawn from a baseline of 56. The same computation as Chapter 0, now done in one pass. Drawn heights: 58 − 56 = 2 and 61 − 56 = 5.

Clause 1 of the contract is violated, a 29x exaggeration. The honest render is bars from zero, where the 5.2% gain looks like what it is.

python — the pocket auditor, 3 lines
def lie_factor(baseline, v1, v2):
    h1, h2 = v1-baseline, v2-baseline
    return ((h2-h1)/h1) / ((v2-v1)/v1)
print(round(lie_factor(56, 58, 61), 1))   # 29.0

Drop any of this lesson's plots onto the contract wheel below and each of the five clauses lights green or red with a one-line reason. Tap a sibling node in the family map to see what it owns.

"The greatest value of a picture is when it forces us to notice what we never expected to see." — John Tukey
— and its greatest danger is when it forces us to conclude what the data never said.