Your tests are green and your model is broken. A content-moderation classifier once lost twelve points of recall for three weeks while every dashboard stayed the color of go. This lesson builds the machinery that would have caught it in an hour — golden sets with statistical gates, drift alarms, data validation, canaries, and automatic rollback — and ends by grading the gate itself, because the gate is just another classifier with its own precision and recall.
It is a Tuesday. A trust-and-safety team at a video platform runs a classifier whose job is to flag policy-violating uploads — graphic violence, hate, the things that get a company on the evening news. On a good day it catches about 90 out of every 100 truly violating videos. That number, the fraction of bad things it catches, is called recall, and the team has spent two years driving it up to 0.90.
On this particular Tuesday, an entirely different team — the ingestion team, who have never heard of the classifier — renames a field in the upload pipeline. The field used to be called audio_transcript. Now it is asr_text. The classifier's feature code still asks for audio_transcript, gets nothing, and does what feature pipelines are built to do: it fills the hole with a default. The transcript feature silently becomes null — imputed to zeros — for the roughly 40% of uploads that have speech.
Nothing crashes. No exception is thrown. The service returns HTTP 200 on every request. Latency is fine. Every unit test is green, because the unit tests test the team's own functions, and those functions ran flawlessly on the garbage they were handed. The model quietly gets worse, one prediction at a time, and no red light comes on anywhere.
This is the defining property of the failures this lesson exists to catch. A regression is any change that makes the system worse for users, whether or not any code changed. In ordinary software, a broken function announces itself: it throws, or it returns a wrong value you can assert against, deterministically, on the first call. A broken model does no such thing. It just gets statistically worse — the distribution of its predictions shifts — and you only see it if you were measuring the distribution.
Let us do the arithmetic of the damage, because the numbers are the whole point. On the 60% of uploads that still have transcripts, recall stays at its healthy 0.90. On the 40% that lost the transcript, recall collapses to 0.60 — the model leaned on that feature. The blended recall — the number an aggregate dashboard actually shows — is the traffic-weighted average of the two slices. Play with the timeline below: inject the change on any day, and watch the blended line fall while the green checkmarks keep smiling.
Each dot is one day's measured recall on a 500-positive labeled slice, with its noise error bar. Press Ship upstream change to inject the silent regression at the current day. Drag affected fraction to change how much traffic loses the feature. Toggle 3σ control band to see what monitoring would have caught — and watch the missed-violations counter climb.
The blended recall lands at 0.78 — a twelve-point fall — and here is the brutal part: that drop was not subtle. Measured against the day-to-day sampling noise of the recall estimate, it was nine standard errors wide. Statistically it was a fire alarm going off in an empty building. Anyone computing daily recall on a labeled slice would have seen it scream on day one. Nobody was computing daily recall. The alarm rang for three weeks into a room with no ears in it.
The reason this class of bug is invisible by default is a mismatch between the failure signal and the monitored signal. The failure signal here is label-dependent quality — recall, which requires knowing the true labels. The signals we monitor by reflex are uptime, error rate, and latency, none of which need labels and none of which moved. Observability for software is simply not observability for models. You built dashboards for the wrong quantity.
The layers differ enormously in how fast they see a regression, and that latency is the whole design tension — the fastest signals are the least direct, and the most direct signal is the slowest. Here is the ladder for Chapter 0's incident:
| Detector | What it watches | Detection latency |
|---|---|---|
| Schema / null-rate check | The feature became null | Minutes — the very first batch |
| Input drift (PSI/KS) | Feature distribution shifted | Hours — first daily scoring |
| Output-score drift | Prediction distribution moved | Hours — label-free, model-aware |
| Daily labeled slice | Recall directly | ~1 day — needs labels |
| Full ground-truth metrics | True recall on all traffic | Days–weeks — labels mature slowly |
The same incident, five detectors, five very different alarm times. The null-rate check would have paged in the first hour; the team was relying (implicitly) on the bottom row.
So the job of this lesson is to convert "someone eventually notices" into "a machine blocks or pages within minutes." Everything downstream is machinery for that conversion. We will build it in layers, defense-in-depth: golden sets with statistical release gates (Ch2), seed-aware tolerance windows so those gates do not flake (Ch3), drift detectors that watch the inputs while labels are still in the mail (Ch4), data-validation suites that ask the blunt question "is this data even legal?" (Ch5), canary and shadow deployments that test on live traffic (Ch6), and performance gates on the latency tail (Ch7). The whole thing culminates in a single reframe (Ch8): the gate itself is a classifier over releases, with its own precision and recall, and you must grade it like one.
This is not a toy scenario invented for a lesson. In September 2025, Anthropic published a postmortem on three overlapping infrastructure bugs — including a context-window routing error and a miscompiled sampling kernel — that silently degraded Claude's response quality for a slice of traffic over August and September. Detection took weeks, at frontier scale, with world-class engineers, precisely because quality regressions are statistically subtle in exactly the way this chapter describes. The Chapter 0 incident is not a straw man; it is the default failure mode of every ML system, and the biggest labs are not immune.
The economics are worth stating plainly, because they justify every hour of gate-building. The cost of a gate is measured in blocked releases and engineer time — annoying but bounded. The cost of no gate is measured in incidents like this one: over three weeks, at 500 violating uploads a day, this bug let roughly 1,260 extra violating videos go live. That is the trade. Let us make it precise.
Setup. Baseline recall 0.90. An upstream change nulls the transcript feature for 40% of uploads; on that affected slice recall falls to 0.60. The platform sees 500 truly violating uploads per day. Question: how big is the blended regression, how many extra violations slip through, and was the drop statistically detectable from a one-day labeled sample?
Step 1 — blended recall. Weight each slice's recall by its share of traffic:
Step 2 — multiply out. 0.60 × 0.90 = 0.54. And 0.40 × 0.60 = 0.24. So blended recall = 0.54 + 0.24 = 0.78.
Step 3 — the drop. 0.90 − 0.78 = 0.12, a twelve-point fall in recall.
Step 4 — missed violations, before. The miss rate is 1 − recall. Before: (1 − 0.90) × 500 = 0.10 × 500 = 50 missed per day.
Step 5 — missed violations, after. (1 − 0.78) × 500 = 0.22 × 500 = 110 missed per day. The extra misses are 110 − 50 = 60 per day. Over 21 days: 60 × 21 = 1,260 extra violating videos live.
Step 6 — detectability. A recall estimate on 500 positives has a binomial standard error. Under the healthy rate p = 0.90:
The drop of 0.12 is 0.12 / 0.0134 = 8.9 standard errors. A single day of labeling 500 positives would have flagged this at a significance level that rounds to impossible-by-chance. The signal was there; the detector was not.
From scratch — simulate the thirty days and find the day a 3σ rule fires:
python import random, math random.seed(0) n_pos = 500 # labeled violating uploads per day base = 0.90 # healthy recall blended = 0.78 # recall after the silent change inject_day = 10 se = math.sqrt(base * (1 - base) / n_pos) # 0.0134 threshold = base - 3 * se # 3-sigma control limit for day in range(30): p = base if day < inject_day else blended hit = sum(random.random() < p for _ in range(n_pos)) # binomial draw est = hit / n_pos fired = " <-- 3-sigma ALARM" if est < threshold else "" print(f"day {day:2d} recall={est:.3f}{fired}")
The alarm fires on day 10 — the very first post-change day — and every day after. Library equivalent — one line to prove the first bad day is not chance:
python from scipy.stats import binomtest # observed ~390 hits out of 500 when true recall is 0.78, tested against p=0.90 print(binomtest(390, 500, p=0.90).pvalue) # p < 1e-15 — same verdict
Same answer: the drop is not noise. The gap between "detectable in one day" and "undetected for three weeks" is the entire gap this lesson closes.
Nothing "broke" in the software sense. The pipeline imputed nulls and kept serving; the model simply lost a signal it depended on for 40% of inputs. That is the defining property of ML regressions — they degrade statistics, not stack traces. Which raises the next question: the transcript rename was a data regression, but data is only one of the surfaces that can rot. Before we build detectors, we need the full map of what can regress. That is Chapter 1.
In Chapter 0 the regression came from data — a renamed field. But data is only one of five distinct places an ML system can rot, and the reason classical CI feels so trustworthy yet catches so little is that it was built to guard exactly one of them.
Here are the five regression surfaces, each with a real way it goes wrong:
MODEL. You retrain on fresher data and the aggregate metric ticks up — but a minority slice (say, non-English uploads) quietly drops three points because the fresh data under-represented it. Nothing in the repository changed; the weights did.
DATA. Chapter 0's null feature. Or a vendor changes a distance field from meters to feet, and every "how far away" feature is now 3.28× too big. The schema still validates; the numbers are lies.
CODE. A preprocessing refactor changes how text is tokenized; a library bump changes the default image interpolation from bilinear to nearest-neighbor. This is the one surface classical CI was designed for — and even here, a dependency's behavior change slips past unit tests that only test your functions.
INFRA. A quantized serving build diverges numerically from the trained checkpoint; a GPU driver update changes a kernel's rounding. The model you ship is not byte-for-byte the model you evaluated.
PROMPT. Someone "improves" a system prompt for an LLM feature and, as a side effect, breaks structured-output parsing for 3% of cases. Zero weight change, zero code diff — just a string edit that alters behavior.
The standard defense arranges tests in a pyramid, cheap and fast at the bottom, expensive and truthful at the top:
The shape matters because of a property specific to ML: the bottom layers are deterministic and can run on every commit in seconds; the top layers are statistical and need samples, time, and sometimes live traffic. You cannot canary every commit — a canary takes days (Ch6 does that math). So the bottom must filter aggressively, catching the cheap majority before the expensive layers ever run.
A subtle but load-bearing distinction lives in what each layer asserts. Traditional tests assert exact behavior: f(2) == 4, pass or fail, no noise. Model tests assert statistical behavior: "accuracy on this set is within noise of baseline." Mix these up — an exact-match assertion on a noisy model metric — and you get a flaky test that fails on luck; loosen a unit test to tolerate noise it does not have, and you get a test that catches nothing. Keep the two kinds of assertion on the two kinds of layer.
It helps to trace each surface to the layer that is supposed to catch it, and its realistic catch probability — because that mapping is what tells you where to invest. A code regression is caught by unit + integration tests with high probability; that surface is well-defended by construction. A data regression is invisible to unit tests (your functions ran fine on the garbage) and to model tests unless the golden set happens to sample the affected slice — only data validation and online monitors reliably see it. A prompt regression is invisible to everything except a golden-set eval that exercises the broken behavior. An infra regression (a quantized build) is invisible until an integration test runs the real serving stack or a canary serves real traffic. Each surface has exactly one or two layers that can see it, and the escape rate of the whole system is set by whichever surface's defending layer is weakest — which is the arithmetic of the worked example below.
| Surface | Layer that catches it | Blind to |
|---|---|---|
| Code | Unit + integration | — (well-defended) |
| Data | Data validation, online monitors | Unit, model (usually) |
| Model | Golden-set gate | Unit, data checks |
| Infra | Integration w/ real serving, canary | Unit, data, model (offline) |
| Prompt | Golden-set gate, shadow | Unit, data, integration |
Read the "blind to" column as the escape routes: a data regression walks straight past unit tests and often past the model gate too.
Play with the pyramid below: fire a regression of each surface type and watch which layer catches it — then disable a layer and watch escapes spike.
Each button drops a regression particle of one surface type at the top. It falls through the five layers; each layer catches it with that layer's per-surface probability (a flash + a tally), or lets it fall. Whatever reaches the bottom lands in ESCAPED. Tap a layer to toggle it off and watch escapes climb. Run 100 releases batch-drops a realistic mix.
Setup. A team ships 100 releases per quarter; historically 8% contain a real regression, spread uniformly across the five surfaces. Their suite's catch probability per surface: code 0.9 (strong unit + integration tests), model 0.8 (golden-set gate), data 0.7 (schema checks), infra 0.6 (staging mirrors prod imperfectly), prompt 0.5 (partial eval coverage). How many regressions reach production per quarter?
Step 1 — miss probability per surface. Miss = 1 − catch: code 0.1, model 0.2, data 0.3, infra 0.4, prompt 0.5.
Step 2 — average miss. Because regressions are uniform across the five surfaces, the average miss probability is the plain mean:
Step 3 — expected true regressions. 100 releases × 0.08 = 8 real regressions per quarter.
Step 4 — expected escapes. 8 × 0.30 = 2.4 regressions reach production per quarter — even with a decent suite. This is not despair; it is a budget you must know.
Step 5 — the weakest layer dominates. Raise prompt coverage from 0.5 to 0.8. New average miss = (0.1 + 0.2 + 0.3 + 0.4 + 0.2) / 5 = 1.2 / 5 = 0.24, so escapes fall to 8 × 0.24 = 1.92. Improving your worst layer bought a bigger cut than any tweak to your best one would.
From scratch — confirm the 2.4 empirically by simulating 10,000 quarters:
python import random random.seed(0) catch = {"code": 0.9, "model": 0.8, "data": 0.7, "infra": 0.6, "prompt": 0.5} surfaces = list(catch) def quarter(): escapes = 0 for _ in range(100): # 100 releases if random.random() < 0.08: # is this one a real regression? s = random.choice(surfaces) # which surface? if random.random() > catch[s]: # suite MISSED it? escapes += 1 return escapes avg = sum(quarter() for _ in range(10_000)) / 10_000 print(round(avg, 2)) # ~2.4
Library equivalent — vectorized with numpy, no loop:
python import numpy as np miss = np.array([0.1, 0.2, 0.3, 0.4, 0.5]) # per-surface miss rates draws = miss[np.random.randint(0, 5, 100_000)] print((np.random.rand(100_000) < draws).mean() * 8) # ~2.4
The regression lives in third-party behavior — invisible to your unit tests and irrelevant to schema (the images are still valid images). A pinned end-to-end integration test with stored golden outputs and a tolerance is the layer designed for "something in the pipeline changed behavior." That is why integration fixtures earn their maintenance cost.
The boundary before we go deeper: this lesson owns what the gates check. How releases physically move — blue/green, feature flags, CI runners — belongs to Testing & Deployment. With the map drawn, we climb to the layer that decides "ship or block": the statistical golden-set gate. That is Chapter 2.
You have a golden set and two models. The candidate scores 84.7%, the baseline 86.2%. Ship or block? It is a trick question — you cannot answer it without the noise. This chapter turns that instinct into a rule.
Start with the object. A golden set is a fixed, versioned, labeled evaluation set that every candidate release must pass. There are two philosophies for building one, and mature teams run both. A curated set is hand-picked: hard cases, past incidents, protected slices — high signal per example, deliberately unrepresentative. A sampled set is random draws from production traffic — representative, so it estimates real-world metrics. Curated cases act as tripwires; the sampled set is the statistical gate.
Why both, and not just one? A curated set alone lies about your aggregate quality, because it over-weights the hard cases you deliberately collected — a model that scores 70% on a set of nightmares might be 95% on real traffic. A sampled set alone misses the rare-but-catastrophic cases: if 0.1% of inputs are the adversarial pattern that caused last quarter's incident, a 4,000-example random sample contains only about four of them, far too few to gate on. So the sampled set answers "is aggregate quality holding?" with statistics, and the curated set answers "did we reintroduce a known failure?" with tripwires. Different questions, no substitution.
Pin down the data flow, because it is where teams cut corners. Each golden example is a frozen (input, label) pair. On every candidate release, the harness runs both the baseline and the candidate over the identical set, producing two boolean arrays — correct/incorrect per example — each of shape (n,). Those two arrays are the only inputs to the gate; raw scores, confidences, and latencies are logged, but the release decision reduces to comparing two length-n boolean vectors. Hold that picture: the gate is a function of two boolean arrays, and everything below is how to compare them without being fooled by noise.
The obvious gate — "block if candidate metric < baseline metric" — is broken in both directions. Eval noise makes two genuinely-equal models disagree by luck, so the naive gate blocks good releases roughly half the time. And a real, small regression can slip under the noise and pass by luck. Comparing two numbers is a coin flip dressed as a decision.
The fix has two parts. First, gate on the confidence interval of the difference, not on the raw numbers. Second — and this is the multiplier — because both models run on the same examples, use a paired design, which is dramatically tighter than comparing two independent evals.
Here is the paired machinery, concretely. On n examples, count b = examples the baseline gets right but the candidate gets wrong, and c = the reverse (candidate right, baseline wrong). Everything both models get right, or both get wrong, cancels — it carries no information about which is better. The entire release decision lives in the disagreements. The estimated difference and its standard error are:
That b + c under the square root is the punchline of pairing: only the disagreements contribute noise. Similar models disagree rarely, so the paired SE is small — which is exactly what lets a modest golden set resolve a one-point margin.
Choose a regression margin δ — the biggest silent drop you are willing to accept. This is a product decision, not a statistical one; say 1 point. Compute the 95% CI for the diff. Then:
Notice what the gate does not do: it does not check whether the CI includes zero. A gate is a one-sided question about the downside, not a two-sided significance test about the point estimate. Drag the sliders below and watch the CI whisker slide across the block line.
Top: a number line of the metric diff, with the block line at −δ; each Evaluate draws a noisy paired eval and renders the diff dot with 95% CI whiskers, PASS if clear of the line, BLOCK if the lower bound crosses it. Run 100 fills the strip below and tallies false blocks (good candidate blocked) vs missed regressions (bad candidate passed). Shrink n and watch the whiskers explode.
Setup. Golden set n = 1000. Baseline gets 862 correct (86.2%). The candidate wins on c = 25 examples the baseline missed, but loses on b = 40 the baseline had right. Policy: block if the 95% CI lower bound is below −δ with δ = 0.01 (1 point). Decide.
Step 1 — candidate accuracy. Candidate correct = 862 − b + c = 862 − 40 + 25 = 847, so 0.847 vs baseline 0.862.
Step 2 — point estimate. diff = (c − b)/n = (25 − 40)/1000 = −15/1000 = −0.015. The candidate looks 1.5 points worse.
Step 3 — only disagreements count. b + c = 40 + 25 = 65 examples decide the release. The other 935 cancel.
Step 4 — standard error.
Step 5 — the 95% CI.
Step 6 — the verdict. The lower bound −0.0308 is below −δ = −0.01, so BLOCK. Note the CI's upper end touches +0.0008 — the candidate might even be better — but the gate's job is to bound the downside, and a 3-point drop is still on the table. You do not ship a coin that might land on a 3-point regression.
Step 7 — sizing check. Could this golden set even police a 1-point margin? To detect a 1-point drop with one-sided α = 0.05 (z = 1.645) and 80% power (z = 0.84) at a 6.5% disagreement rate:
You need about 4,000 examples. A 1,000-example golden set cannot reliably police a 1-point margin — its whiskers are too wide. The disagreement rate is the lever people forget: similar models disagree rarely, and low disagreement helps the paired design.
Step 8 — the under-powered flip. To feel why sizing matters, shrink the same experiment to n = 250 with the same rates (b = 10, c = 6). Now diff = (6 − 10)/250 = −0.016, and SE = √16 / 250 = 4 / 250 = 0.016. The 95% CI is −0.016 ± 1.96 × 0.016 = [−0.047, +0.015] — wide enough that a 4.7-point regression cannot be excluded. On this set, almost every candidate blocks, regardless of quality: an under-powered golden set does not just miss regressions, it also blocks good releases, because its CI is too wide to clear the margin either way. The sim's n slider makes this visceral — drag n down and watch nearly every candidate turn red.
From scratch — the whole gate in twelve lines:
python import math def paired_gate(base_correct, cand_correct, delta=0.01, z=1.96): # base_correct, cand_correct: boolean arrays over the SAME examples n = len(base_correct) b = sum(bc and not cc for bc, cc in zip(base_correct, cand_correct)) c = sum(cc and not bc for bc, cc in zip(base_correct, cand_correct)) diff = (c - b) / n se = math.sqrt(b + c) / n lo, hi = diff - z * se, diff + z * se return {"diff": diff, "se": se, "ci": (lo, hi), "verdict": "BLOCK" if lo < -delta else "PASS"} def required_n(delta, p_disagree, z_a=1.645, z_b=0.84): return math.ceil((z_a + z_b)**2 * p_disagree / delta**2) print(required_n(0.01, 0.065)) # 4014
Library equivalent — the paired test is McNemar's test:
python from statsmodels.stats.contingency_tables import mcnemar # table = [[both wrong-agree], [b, c]] disagreement cells result = mcnemar([[0, 40], [25, 0]], exact=False, correction=False) print(result.pvalue) # the paired significance; CI via scipy.stats.norm.ppf
A golden set is code, and it rots like code. Three rules keep it honest. Version it: hash the set, log the hash with every eval, so a "regression" that is really a set change is diagnosable. Refresh it on a schedule: production drifts away from any frozen set, so a two-year-old golden set measures a world that no longer exists. Refresh without leaking: never add examples selected because the current candidate fails them — that manufactures a gate biased against exactly the model under review — and never let golden examples enter training data. Quarantine windows and held-out refresh pools are the standard tools.
The curated tripwire set plays by a different rule. If the exact bug from incident #421 reproduces, you block — regardless of aggregate stats. These are zero-tolerance cases, each documented with its incident link so a future engineer knows why the tripwire exists.
The gate bounds the downside: block when the data cannot exclude a drop bigger than δ. Here the CI admits drops as large as 1.8 points, beyond the tolerated 1.0. "CI includes zero" is the wrong test entirely — a gate is a non-inferiority check on the lower bound, not a significance test on the point estimate.
But there is a crack in everything we just built. It assumed the eval gives one number. Run the same model twice and you get two different numbers — and a gate that does not know its own run-to-run noise will flake. That is Chapter 3.
Your gate blocked a release on Tuesday. Someone clicked "retry" and it passed on Wednesday. Same model, same golden set, same code. What happened? The answer — and why it silently destroys gates — is this chapter.
Run-to-run variation is not a bug in your harness; it is physics of the stack, and it comes from at least four mechanistic sources.
Training seeds. Weight initialization, data shuffling, and dropout all consume randomness. Retraining "the same" model with a new seed yields a genuinely different model — often 1–2 points apart on small benchmarks.
GPU nondeterminism. Floating-point addition is not associative: (a + b) + c ≠ a + (b + c) in the last bits. A GPU's atomicAdd reduces in whatever order threads finish, so the sum changes run to run; cuDNN's autotuner may pick a different kernel; even inference can differ across batch sizes.
Sampling temperature. For generative models, temperature > 0 means the eval draws a random output per prompt. Two eval runs literally sample different completions.
Eval-harness slop. Unpinned dataset ordering, wall-clock-dependent timeouts, and a judge model whose provider silently updates it — all inject variation that has nothing to do with the candidate.
Two variances get conflated and must not be. Training variance: retrain with a new seed, get a different model (1–2 points). Inference/eval variance: same weights, rerun the eval, get a different number. Your gate must tolerate the second and often the first; conflating them makes you either flaky or blind.
The batch-composition source deserves a mechanistic look, because it is the one that surprises people who "pinned everything." When a serving engine batches your prompt together with other concurrent requests, the matrix multiplications inside the model are computed over the whole batch, and the reduction (the summation of many products) is split across GPU threads whose completion order depends on the batch's exact shape. Different batch shapes → different reduction orders → different last-bit rounding in the logits → occasionally a different argmax → a different token. Your prompt did not change; the other traffic sharing its batch changed, and that leaked into your output. This is why the same eval run at 2 a.m. (empty servers) and at noon (full batches) can differ.
Here is why a flaky gate is worse than no gate. Set a tolerance narrower than the run noise, and the gate blocks good releases at random. Engineers learn that clicking "retry" works. The gate loses all authority — it becomes a ritual, not a check. And then, the day a real regression comes, it sails through on a lucky retry, because the team has already been trained to retry until green. A flaky gate does not just fail to help; it actively erodes the discipline that would have caught the problem.
The fix is empirical, not theoretical: measure your run-to-run standard deviation by evaluating the same artifact k times, then set the tolerance window from that measured sd — never from vibes. Play with the strip below: crank the nondeterminism slider and watch single runs flip a fixed-tolerance gate at random while the k-run mean stays calm.
Each Run eval drops a score dot drawn from a hidden true mean plus noise (the nondeterminism slider). A running mean line and shrinking ±2 SE band update live. Drag the tolerance window edges: the readout shows how often a single run would cross the window vs the k-run mean. Toggle reveal true mean to see the truth the noise is hiding.
Setup. You evaluate the same candidate five times with different seeds: 0.812, 0.798, 0.805, 0.821, 0.784. The baseline's official number is 0.810. (a) Summarize the run distribution. (b) Is the candidate "below baseline"? (c) How many runs for the mean's SE to reach 0.005?
Step 1 — the mean. (0.812 + 0.798 + 0.805 + 0.821 + 0.784) / 5 = 4.020 / 5 = 0.804.
Step 2 — deviations from the mean. +0.008, −0.006, +0.001, +0.017, −0.020.
Step 3 — squared deviations. 0.000064, 0.000036, 0.000001, 0.000289, 0.000400; sum = 0.000790.
Step 4 — sample variance and sd. Divide by (5 − 1) for the unbiased estimate:
That is about 1.4 points of pure run noise — no model changed.
Step 5 — SE of the mean and a t-interval. SE = sd / √5 = 0.01405 / 2.236 = 0.00628. With 4 degrees of freedom, t = 2.776:
Step 6 — is it below baseline? The baseline 0.810 sits comfortably inside [0.787, 0.821]. The five runs cannot distinguish this candidate from the baseline. And notice: a single unlucky run (0.784) would have looked like a 2.6-point regression — pure noise, and a naive single-run gate would have blocked it.
Step 7 — runs needed for SE = 0.005. Since SE = sd / √k, solve for k:
And beware diminishing returns: doubling k only shrinks SE by √2 ≈ 1.4×. Past a point, grow the golden set (Ch2's lever) instead — example noise and seed noise add, so the cheaper win might be more examples, not more runs.
From scratch — run stats by hand plus a needed-runs helper:
python import math def run_stats(scores): k = len(scores) mean = sum(scores) / k ss = sum((s - mean)**2 for s in scores) # sum of squared deviations var = ss / (k - 1) # ddof=1 sd = math.sqrt(var) se = sd / math.sqrt(k) return mean, sd, se def needed_runs(sd, target_se): return math.ceil((sd / target_se)**2) scores = [0.812, 0.798, 0.805, 0.821, 0.784] print(run_stats(scores)) # (0.804, 0.01405, 0.00628) print(needed_runs(0.014053, 0.005)) # 8
Library equivalent — test the run distribution against the baseline in one line:
python from scipy.stats import ttest_1samp print(ttest_1samp(scores, 0.810).pvalue) # large p — indistinguishable from baseline
Operationally: gate on the mean of k runs against a window of ±z × sd/√k; pin everything pinnable (temperature 0 where faithful to prod, fixed dataset order, pinned judge version, fixed batch size); and log the seed and every per-run score, so a blocked release shows its whole distribution, not one number. A gate that reports "0.784, BLOCKED" without the other four runs is hiding the very information needed to overrule it.
With noise nearly 3× the tolerance, identical-quality candidates routinely land outside the window by luck. Teams respond by retrying, converting the gate into a ritual — and once retry-until-green is culture, a real regression just needs one lucky run to ship. Tolerance windows must come from measured run variance (and shrink by averaging k runs), never set narrower than the noise.
So far every gate runs offline, on a labeled golden set. But Chapter 0's incident was invisible offline — the labels arrived weeks late. What is visible immediately is the inputs. Watching them is Chapter 4.
Labels arrive days or weeks late — fraud is confirmed by chargebacks, moderation by appeals. But the inputs are visible right now. Drift detection is how you get tomorrow's incident report today: it turns a lagging quality signal into a leading input signal.
Chapter 0's regression was detectable from ground truth only because someone could label a daily slice. Most systems cannot. But the feature distributions shifted the instant the upstream change landed. Monitoring inputs is how you see that shift without waiting for labels.
Not all drift is the same, and the differences decide which detector even can see it. Write the joint distribution as p(x, y) = p(x) × p(y | x).
Covariate drift: p(x) changes, p(y | x) intact. A marketing campaign brings younger users; the inputs shift, but the relationship between inputs and labels holds — the model may still be fine.
Concept drift: p(y | x) changes. Fraudsters adapt to your rules, so the same transaction patterns that used to be legitimate are now fraud. The inputs barely move; their meaning changed. This is invisible to input monitoring — only labels reveal it.
Label/prior drift: p(y) changes. The violation base rate doubles during an election. The mix of outcomes shifts even if individual inputs look normal.
The reason to bother with input monitoring at all is the timing gap, and it is worth quantifying. In a fraud system, a labeled outcome (a confirmed chargeback) matures in roughly three weeks. If you only measured quality from labels, a regression that landed today would surface in your metrics three weeks from now — three weeks of damage before the first data point even exists. The input distribution, by contrast, shifted today, the instant the bad batch arrived. Input monitoring does not measure quality; it buys you a three-week head start on the hypothesis that quality is about to fall. That head start is the entire value proposition, and it is why every mature stack pairs a fast label-free detector with a slow ground-truth loop.
The Population Stability Index (PSI) is the workhorse for binned tabular features. The recipe: bin the baseline distribution into deciles (10 bins, each holding 10% of baseline), measure today's fraction in each bin, and sum a per-bin surprise:
It is a symmetrized KL divergence. Each bin contributes a nonnegative amount, so you can rank which bins moved — that per-bin decomposition is the diagnostic gold, telling you not just that the distribution drifted but where. Drag the histogram below and watch PSI, KS, and KL respond, with the per-bin contribution bars showing you exactly which bins are driving it.
The outlined histogram is the frozen baseline (10 deciles); the filled one is live — morph it with shift and spread. Below, per-bin PSI contribution bars and three gauges: PSI, KS D, KL, each marked with the folklore thresholds. Replay incident snaps to Chapter 0's null-spike and shows PSI exploding past 0.25.
Setup. A model-score feature has 10 equal-population baseline bins (expected fraction 0.10 each). Today's fractions: [0.06, 0.07, 0.09, 0.10, 0.11, 0.12, 0.13, 0.12, 0.11, 0.09] — a mild shift toward higher scores. Compute PSI bin by bin, the KS statistic on the binned CDFs, and KL(actual ‖ expected). Compare 5,000 samples/side against the KS critical value.
Step 1 — PSI, first bins. Each term is (a − e) × ln(a/e).
Step 2 — middle bins. Bin 4: (0) × ln(1.0) = 0. Bin 5: (0.01) × ln(1.1) = 0.01 × 0.09531 = 0.00095. Bin 6: (0.02) × ln(1.2) = 0.02 × 0.18232 = 0.00365. Bin 7: (0.03) × ln(1.3) = 0.03 × 0.26236 = 0.00787.
Step 3 — tail bins. Bin 8: (0.02) × ln(1.2) = 0.00365. Bin 9: (0.01) × ln(1.1) = 0.00095. Bin 10: (−0.01) × ln(0.9) = 0.00105.
Step 4 — sum.
Folklore thresholds — < 0.10 stable, 0.10–0.25 investigate, > 0.25 significant — call this "stable." But bins 1–2 contribute 62% of the score: the low-score mass is evaporating. The total says relax; the decomposition says look closer.
Step 5 — the KS test. The Kolmogorov–Smirnov statistic D is the maximum vertical gap between the two empirical CDFs. On the binned CDFs: cumulative actual = [0.06, 0.13, 0.22, 0.32, 0.43, 0.55, 0.68, 0.80, 0.91, 1.00]; cumulative expected = [0.10, 0.20, …, 1.00]; absolute gaps = [0.04, 0.07, 0.08, 0.08, 0.07, 0.05, 0.02, 0.00, 0.01, 0.00]; so D = 0.08.
Step 6 — the KS trap at scale. The critical value at α = 0.05 with n = m = 5000:
D = 0.08 ≫ 0.0272 — statistically "unmistakable," even though PSI called it stable. Same data, two verdicts: PSI measures magnitude, KS measures detectability, and at n = 5000 you can detect shifts far smaller than ones worth acting on. This is why, at scale (n in the millions), you gate on effect size (D itself, e.g. D > 0.05), never on the KS p-value — the p-value declares microscopic, harmless shifts "significant."
Step 7 — KL divergence. The third lens, KL(actual ‖ expected) = Σ a × ln(a/e), the foundation PSI is built from (PSI sums both KL directions). On these bins it equals 0.0243 — same ballpark, same story. (KL is asymmetric and blows up on empty expected bins, so add epsilon smoothing in code.)
From scratch — PSI with per-bin contributions plus binned KS:
python import numpy as np def psi(expected, actual, eps=1e-6): e = np.clip(np.asarray(expected, float), eps, None) a = np.clip(np.asarray(actual, float), eps, None) contrib = (a - e) * np.log(a / e) # per-bin, for ranking return float(contrib.sum()), contrib def binned_ks(expected, actual): return float(np.max(np.abs(np.cumsum(actual) - np.cumsum(expected)))) e = np.full(10, 0.1) a = np.array([0.06, 0.07, 0.09, 0.10, 0.11, 0.12, 0.13, 0.12, 0.11, 0.09]) total, contrib = psi(e, a) print(round(total, 4), round(binned_ks(e, a), 2)) # 0.0503 0.08
Library equivalent — exact two-sample KS with a p-value, and KL:
python from scipy.stats import ks_2samp, entropy # raw samples, not binned fractions print(ks_2samp(baseline_sample, today_sample)) # D and p print(entropy(a, e)) # KL(a || e) = 0.0243
You cannot bin a sentence or an image. Embed it with a frozen encoder, then monitor the embedding distribution — centroid drift (mean embedding distance between windows), per-dimension PSI on projections, or MMD. The encoder must not change: swap it and every drift score jumps for artifact reasons, not real ones. Also monitor the output distribution (prediction scores) — it is free, model-aware, and often the first thing to move.
A sane alerting policy: score every feature daily; alert on (a) any feature crossing its calibrated red line or (b) a jump of many multiples of its own historical week-to-week noise; and rank alerts by PSI × feature importance, so the top of the page reads "important feature moved a lot," not "noisy junk feature wiggled." Drift alerts are hypotheses for a human, not automatic rollbacks — that trigger belongs to canary metrics (Ch6).
p(y | x) changed while p(x) stayed nearly fixed: the same inputs now carry different labels. Input-distribution monitors see nothing, and output-score monitoring may barely move. This is exactly why the monitoring stack needs a lagging ground-truth loop alongside the fast label-free detectors — each covers the other's blind spot.
Drift asks "did the distribution move?" There is a blunter question to ask first — "is this data even legal?" That is data validation, the cheapest layer of all, and it is Chapter 5.
Drift detection (Ch4) tells you a distribution moved. Data validation asks a blunter question first: is this data even legal? Data validation is unit testing for data — a declarative suite of expectations that runs on every batch and blocks bad data before it reaches training or serving.
This is the cheapest, highest-leverage layer of the pyramid. Chapter 0's incident was a null-rate spike — a five-line check would have caught it in the first hour, no statistics required. Most celebrated "model regressions" are data-quality failures wearing a trench coat.
Each expectation guards a specific way data goes bad, and each has a concrete violation story:
SCHEMA: column exists, type matches, enum values are in the allowed set. The Chapter 0 rename lives here.
RANGE: age in [0, 120]; a meters-to-feet vendor change makes every distance 3.28× too big while every value stays "valid."
NULL-RATE: a feature is nullable but historically 2% null — today it is 27%.
FRESHNESS: the feature-store partition landed late; you are serving yesterday's features and every "recency" signal is stale.
VOLUME: row count within an expected band; a dedup bug halves the table.
CROSS-FIELD: end_time ≥ start_time; refund ≤ purchase.
Here is what separates a toy check from a production check. "Null rate is 2%" is a rate, and today's sample estimate wobbles around it. A hard rule like "fail if > 2%" fires constantly on noise — roughly half of healthy batches sit above the mean of a noisy quantity. The correct check is a tolerance band derived from binomial sampling noise. Play with the gauntlet below: crank the tolerance too tight and watch the false-alarm ticker climb.
Row particles stream through four gates — Schema, Range, Nulls, Freshness — into the model box. Corruption sliders inject bad rows; each bad row bounces off the first gate that catches it (color-coded, with tallies), unless you toggle that gate off — then corrupt rows reach the model, which reddens as its health drains. Crank the tolerance down and the false-alarm ticker shows the cost of over-tight bands.
Setup. A feature's historical null rate is 2.0%. Today's batch: 137 nulls in 5,000 rows. Policy: page at 3σ. (a) Is this an alarm? (b) With 200 such checks per day at 3σ, how many false alarms per day from noise alone?
Step 1 — observed rate. 137 / 5000 = 0.0274 (2.74%).
Step 2 — sampling SE under the baseline. Binomial SE with p = 0.02:
Step 3 — the z-score.
Step 4 — verdict. 3.74 > 3, so PAGE. The excess is 37 nulls beyond the expected 100 — a real shift, not batch-to-batch wobble. A fixed "> 2%" rule would have paged on this and on every healthy 2.3% batch.
Step 5 — the false-alarm budget. A two-sided 3σ check false-fires with probability ≈ 0.0027. Over 200 independent checks:
Step 6 — the consequence. At 200 checks you buy one noise-page every ~2 days, forever. That is the multiple-testing tax: the more checks you run, the tighter each per-check threshold must be (or the more checks route to a digest) to keep on-call trust. Tiering — page-worthy checks at 4σ (per-check false rate ~0.0000633), 3σ for digest-level warnings — is not bureaucracy; it is what keeps humans answering the pager.
From scratch — a mini expectations engine in ~30 lines:
python import math SUITE = [ {"col": "device_age", "check": "null_rate", "baseline": 0.02, "sigma": 3, "severity": "page"}, # ... more expectation dicts ] def validate(batch, suite): report = [] for exp in suite: col = batch[exp["col"]] n = len(col) if exp["check"] == "null_rate": rate = sum(v is None for v in col) / n p = exp["baseline"] se = math.sqrt(p * (1 - p) / n) z = (rate - p) / se hit = abs(z) > exp["sigma"] status = exp["severity"] if hit else "pass" report.append({"col": exp["col"], "z": round(z, 2), "status": status}) return report # 137 nulls in 5000 -> z = 3.74 -> "page"
Library equivalent — the same declaratively with great_expectations or pandera:
python # great_expectations: # validator.expect_column_values_to_not_be_null("device_age", mostly=0.975) import pandera as pa schema = pa.DataFrameSchema({ "device_age": pa.Column(float, pa.Check.in_range(0, 120), nullable=True), }) schema.validate(df) # raises on the first expectation a batch violates
Seeing the engine is 30 lines demystifies the vendor tools: the value is in the expectation library and its calibration, not the engine.
The forgotten half is label-quality auditing. Your golden sets and training data have wrong labels at some base rate, and label errors regress too — a guideline change, a new annotation vendor, an ontology migration. Standing audits: inter-annotator agreement on a re-labeled sample, per-annotator agreement drift, and a "model-disagrees-with-label" queue sorted by confidence — the cheapest wrong-label miner in practice.
The math here matters because label noise silently caps your gate. Suppose your golden set has a 3% label-error rate. Then a perfect model — one that gets every input truly right — scores only 97% on that set, because 3% of the "wrong" answers are actually the labels being wrong. Worse for regression testing: when labels shift (a new vendor relabels a slice differently), the golden set's measured accuracy moves even though no model changed — a pure label regression that your gate will attribute to the candidate. This is why the model-disagrees-with-label queue is the highest-leverage audit: sort by cases where a high-confidence model disagrees with the label, and the top of that queue is where your labels are wrong far more often than your model is. Fixing those relabels the set, tightens the ceiling, and removes a source of phantom regressions.
The same expectation library runs at three enforcement points: at ingestion (block the pipeline, quarantine the batch), before training (refuse to train on a corrupt snapshot — a bad model trained is far more expensive than a batch delayed), and at serving (per-request feature sanity with a fallback path: impute-and-log, route to a simpler model, or degrade gracefully).
The failure posture differs from a release gate. A release gate blocks and waits; data validation needs a quarantine path — hold the bad partition, alert the owner, serve from the last good snapshot — and never let "temporarily skip the check" become permanent. Every skipped check gets an expiry date.
Each check false-fires with probability ~0.0027 (two-sided 3σ), so 200 checks generate 0.54 per day — roughly 3.8 per week on perfectly healthy data. This is the multiple-testing tax: more checks means a tighter per-check threshold (or more digests) to keep on-call trust.
Every gate so far — golden sets, drift, data checks — runs before deployment. But offline eval is a rehearsal. Some regressions only exist on the live stage, and catching those is Chapter 6.
Every gate so far ran before deployment. But offline eval is a rehearsal — some regressions only exist on the live stage. This chapter is the final gate: it runs on real traffic.
Four concrete escape routes get past even a perfect offline suite. Serving-stack divergence: the quantized production build differs from the evaluated checkpoint. Feature skew: online features are computed by different code than offline training features. Unseen traffic: new locales, adversarial users, bursty distributions the golden set never contained. System effects: latency under real load changes user behavior itself, which no offline eval can simulate.
Shadow mode is free of user risk: the candidate receives a mirrored copy of live traffic, and its predictions are logged but never served. What it buys you: crash and latency behavior under real load, a prediction-diff against the incumbent on identical inputs, and drift-free comparison (same traffic, same moment). What it cannot see: anything that depends on the model's outputs affecting the world — user reactions, feedback loops, downstream systems.
The diff dashboard is shadow mode's payoff. On N mirrored requests: what fraction of predictions changed? How do the changed-prediction score distributions compare? Which slices changed most? A candidate that "improves the golden set by 1 point" but flips 30% of live predictions deserves suspicion — golden sets are small, and the flip mass is where unknown regressions hide.
A canary serves the candidate to a small random slice for real. Three design variables: the split (random by user, not by request — per-request randomization contaminates sessions and underestimates user-level effects), the stages (1% → 5% → 25% → 100%, with a soak time each), and the guardrail metrics (a small pre-registered set: a primary quality proxy, error rate, latency p99, and 2–3 business guardrails). Drive the console below and watch the guardrail band narrow as samples accumulate — or the error klaxon fire instantly.
Dwell on the split, because it is the mistake that quietly ruins canary numbers. If you randomize per request, a single user's session gets served the incumbent on one page and the candidate on the next — their experience is a blend, so any effect that depends on session coherence (a recommendation that shapes the next click, a conversation that builds context) gets averaged toward zero. You will conclude "no effect" from a change that a user-consistent rollout would have shown clearly. Randomize by a stable user key (hash the user id into buckets) so each user lives entirely in one arm for the whole experiment. The unit of your analysis and the unit of your randomization must match, or your CI is measuring the wrong thing.
Request dots fork to the incumbent (big pipe) and the canary (thin pipe, width = canary % slider). Below: the guardrail diff (canary minus control) as a dot with a band that narrows as samples accumulate, and a fast error ticker. The auto-rollback klaxon fires the moment the band excludes the pre-registered tolerance or the error ticker spikes. Drag true effect negative and watch how long detection takes.
Setup. 200,000 requests/day; canary takes 5% (10,000/day). Baseline conversion 8.0%. You must catch a 10% relative drop (to 7.2%) with one-sided α = 0.05 and 80% power. (a) Users per arm? (b) Soak days? (c) Errors: baseline 0.1% means ~10 expected errors in the first 10k canary requests — you observe 50. React?
Step 1 — the effect. p1 = 0.080, p2 = 0.072, absolute Δ = 0.008.
Step 2 — variance terms. p1(1−p1) = 0.08 × 0.92 = 0.0736; p2(1−p2) = 0.072 × 0.928 = 0.0668. Sum = 0.1404.
Step 3 — the z multiplier. (z0.05 + z0.20)² = (1.645 + 0.84)² = 2.485² = 6.175.
Step 4 — users per arm.
Step 5 — soak days. 13,549 / 10,000 per day = 1.35 days minimum before this quality guardrail even has 80% power. A 1% canary (2,000/day) would need ~6.8 days — small canaries are slow canaries. (The full power / sequential-testing / CUPED machinery lives in experiment-design; here we own the release-safety application.)
Step 6 — the fast guardrail. Expected errors λ = 10, observed 50. Poisson z:
This is not a statistics question anymore. Automatic rollback fires immediately — no human approval belongs in that loop.
From scratch — canary power and a rollback monitor:
python import math def canary_power(p_base, rel_drop, per_day, alpha=1.645, power=0.84): p2 = p_base * (1 - rel_drop) d = p_base - p2 v = p_base * (1 - p_base) + p2 * (1 - p2) n = (alpha + power)**2 * v / d**2 return math.ceil(n), round(n / per_day, 2) print(canary_power(0.08, 0.10, 10_000)) # (13549, 1.35) def rollback_monitor(stream, lam=10, z_stop=4): # stream yields per-minute error counts for errors in stream: z = (errors - lam) / math.sqrt(lam) if z > z_stop: yield "ROLLBACK"; return yield "ok"
Library equivalent — the same n in one call:
python from statsmodels.stats.proportion import samplesize_proportions_2indep_onetail print(samplesize_proportions_2indep_onetail( diff=-0.008, prop2=0.08, power=0.8, alpha=0.05)) # ~13,549
Guardrails split by speed. Fast guardrails — error rate, timeout rate, crash loops — are Poisson-style counts where a 5× spike is detectable in minutes; rollback on these is automatic. Slow guardrails — quality and business metrics that need the power math above — page a human with the dashboard.
The cultural core is pre-registration: rollback criteria are written before the canary starts — "roll back if p99 > 300ms for 15 minutes, or error-rate z > 4, or conversion drops > 1 point with 95% confidence." Deciding criteria during an incident invites sunk-cost rationalization ("let's wait one more hour, it might recover").
Three hours of a 5% slice is far below the ~13.5k-per-arm requirement for even a large effect; the CI [−2.1, +1.3] says "no information yet." Acting on the point estimate now is exactly the noise-chasing that gates exist to prevent. The disciplined move: pre-registered fast guardrails protect you meanwhile, and the quality guardrail waits for its powered sample.
There is one more regression surface hiding in plain sight. A model can be more accurate and still be a regression — because it answers in 900 milliseconds instead of 40. Latency is a distribution, and gating it is Chapter 7.
A model can be more accurate and still be a regression — because it answers in 900 milliseconds instead of 40. Performance is a first-class regression surface, and it hides in a place the mean cannot see.
A quantization change, a longer prompt template, a bigger model, a new retrieval step — each can pass every quality gate while quietly doubling latency or tripling cost. Users experience the slowest interactions disproportionately; finance experiences cost-per-request multiplicatively at scale.
Suppose 98% of requests finish in 40ms and 2% take 900ms. The mean is 0.98 × 40 + 0.02 × 900 = 39.2 + 18.0 = 57.2ms — comfortably under a 100ms SLO. And yet one in fifty users waits nearly a second. The mean is dominated by the happy path; the tail is a different code path — cache misses, retries, cold starts, GC pauses, long generations. Tails are not noise. Drag the slow mode below and watch the mean stay calm while the tail eats your users.
A live latency histogram from a fast mode plus a draggable slow mode. Marker lines for mean, p50, p95, p99 slide as you drag the sliders; the draggable SLO line reddens the violating tail and shows "users affected per 10,000." Toggle fan-out to add the 1−(1−q)k amplification readout.
A percentile is the value below which a given fraction of the sample falls. The nearest-rank p-th percentile: sort the sample, then take the value at index ⌈(p/100) × n⌉. Walk a 20-request example: p95 = 19th sorted value, p99 = 20th — the tail-heavy indices are where the pain lives.
method='higher') or your perf gate flakes.Here is why p99 matters even if you "only care about typical." A request that fans out to 10 backends waits for the slowest. If each backend is slow (worse than its own p99) just 1% of the time, the parent is slow on 1 − 0.9910 = 9.6% of requests. Backend p99 becomes nearly the frontend p90. LLM pipelines with retrieval + rerank + generation are exactly such chains — the tail compounds down the pipe.
And percentile estimates are themselves noisy: a p99 from 1,000 requests rests on the 10 slowest observations, so run-to-run wobble is large (Ch3's lesson, now on the tail). Perf gates need warmed-up runs, fixed load profiles, enough requests that the gated percentile has stable footing, and tolerance bands measured from repeat runs.
Setup. (a) 98% of requests at 40ms, 2% at 900ms; SLO "mean under 100ms." Evaluate the blind spot. (b) 20-request sample [12,14,15,15,16,17,18,18,19,20,21,22,23,25,27,30,34,41,58,120] ms, sorted — compute mean, nearest-rank p95/p99, and numpy's default. (c) With 10-way fan-out, what fraction of frontend requests exceed a backend's p99?
Step 1 (a) — the bimodal mean. 0.98 × 40 + 0.02 × 900 = 39.2 + 18.0 = 57.2ms. "Mean < 100ms" passes with 43ms of headroom — while 1 in 50 requests takes 900ms. The mean SLO is blind to a tail that 2% of every session hits.
Step 2 (b) — sample mean. The values sum to 565, so 565 / 20 = 28.25ms.
Step 3 — nearest-rank p95. index = ⌈0.95 × 20⌉ = ⌈19.0⌉ = 19th value = 58ms.
Step 4 — nearest-rank p99. index = ⌈0.99 × 20⌉ = ⌈19.8⌉ = 20th value = 120ms.
Step 5 — numpy's default (linear interpolation). position = 0.95 × (20 − 1) = 18.05 (0-indexed), so p95 = 58 + 0.05 × (120 − 58) = 58 + 3.1 = 61.1ms. Two legitimate definitions, 3.1ms apart on the same data — a 5%-band gate flips if the method silently changes. Pin it.
Step 6 (c) — fan-out amplification.
About 9.6% of fan-out requests are slower than a single backend's p99. The tail compounds.
From scratch — pinned-definition percentile and a perf gate:
python import math def percentile_nearest_rank(vals, p): s = sorted(vals) idx = math.ceil(p / 100 * len(s)) # 1-based nearest rank return s[min(idx, len(s)) - 1] def perf_gate(cand, base, pcts=(95, 99), band=0.10): for p in pcts: cp = percentile_nearest_rank(cand, p) bp = percentile_nearest_rank(base, p) if cp > bp * (1 + band): # relative regression beyond band return f"BLOCK: p{p} {cp} > {bp*(1+band):.0f}" return "PASS" def fanout_tail(q, k): return 1 - (1 - q)**k a = [12,14,15,15,16,17,18,18,19,20,21,22,23,25,27,30,34,41,58,120] print(percentile_nearest_rank(a, 95), percentile_nearest_rank(a, 99)) # 58 120 print(round(fanout_tail(0.01, 10), 4)) # 0.0956
Library equivalent — the explicit method argument is the gate-stability fix:
python import numpy as np print(np.percentile(a, [50, 95, 99], method="higher")) # nearest-rank: 20, 58, 120 print(np.percentile(a, 95)) # default linear: 61.1
Latency: p95/p99 under a pinned load profile on pinned hardware — gate on both absolute SLO breach and relative regression vs baseline. Throughput: max sustainable QPS at the latency SLO — degrades invisibly if unmeasured. Cost: tokens or GPU-seconds per request × price. For LLMs, output-length regressions are cost regressions: a prompt change that makes answers 30% longer is a 30% serving-cost increase that no per-token latency gate notices. And LLM latency decomposes — time-to-first-token (perceived responsiveness), inter-token throughput (streaming smoothness), total generation time (cost proxy) — and a KV-cache change can improve one while regressing another, so gate them separately.
The output-length cost trap deserves a worked number, because it is the sneakiest perf regression — it hides behind a faster per-token latency. Suppose a "helpfulness" prompt tweak raises the mean answer from 180 to 240 output tokens, a 33% increase. At $15 per million output tokens and 2 million requests per day, the daily output cost was 2M × 180 × $15/1M = $5,400/day; after the tweak it is 2M × 240 × $15/1M = $7,200/day — an extra $1,800/day, ~$657k/year, from a prompt edit that made every individual token render faster. A latency-only gate sees the faster tokens and passes it. Only a gate on the output-length distribution catches it. This is why cost gating needs the length distribution, not just per-token speed — and why "the model got more helpful" and "the model got more expensive" are frequently the same commit.
The frontend is slow if any backend is slow: P = 1 − 0.9910 = 0.0956. (The "10 × 1% = 10%" shortcut is close only because the per-backend rate is small.) A backend's p99 becomes roughly the frontend's p90, which is why deep pipelines must gate backend tails far tighter than the frontend SLO suggests.
We have built seven layers of gates. Now step behind the glass and ask the question that makes the whole lesson click: how good is the gate itself? That is Chapter 8, the showcase.
You have built gates all lesson. Now step behind the glass: run a quarter's worth of releases through your gate and grade the gate. The reframe that makes everything click is this — the gate is itself a binary classifier over releases.
Every release is truly GOOD or truly REGRESSED (hidden from you), and the gate PASSES or BLOCKS it (observed). That is a 2×2 confusion matrix with "regression" as the positive class:
| Gate BLOCKS | Gate PASSES | |
|---|---|---|
| Truly regressed | True positive — the save | False negative — the incident |
| Truly good | False positive — the velocity tax | True negative — business as usual |
So the gate has its own metrics, in exactly the vocabulary of metric-design. Gate recall = fraction of true regressions blocked (1 − recall = the incident escape rate). Gate precision = fraction of blocks that were justified (1 − precision = the fraction of blocks that were noise — the thing that burns engineer trust). These two are in tension through one knob: strictness (the δ margin and confidence level from Ch2, the tolerance window from Ch3, the canary soak from Ch6).
If candidate quality diffs were measured exactly, a threshold at zero would be a perfect gate — no overlap, no error. But eval noise smears the good and regressed populations into overlapping distributions of measured diffs. Now any threshold misclassifies some of each. The gate's ROC curve is literally the two overlapping Gaussians of signal-detection theory, and golden-set size n controls the overlap width: bigger n → narrower noise → better ROC. That is the only free lunch in the room. This is the showcase — run releases through and watch it.
Make the overlap concrete. Good releases cluster their measured diff around 0; regressed releases cluster theirs around, say, −2 points. If the eval noise sd is 0.6 points (a well-powered golden set), those two bells barely touch — a threshold at −1 cleanly separates them, and the gate is nearly perfect.
But shrink the golden set until the noise sd is 1.5 points, and the two bells overlap heavily: a genuinely good release routinely measures −1.8 (blocked, a false positive) while a real regression routinely measures −0.5 (shipped, a false negative). The same threshold now makes both kinds of error, and no amount of moving it fixes both — sliding right catches more regressions but blocks more good releases, and vice versa. You are trapped on one ROC curve. The only escape is to un-overlap the bells, which means shrinking the noise, which means a bigger n. Watch this exact dynamic by dragging the n slider in the sim: the two curves squeeze together or pull apart, and the confusion matrix follows.
A conveyor of release capsules, each with hidden true quality (good vs regressed, mix = base rate) plus measured noise (width = golden-set n). Capsules hit the strictness line and route to SHIPPED or BLOCKED, then flash their true color (teal good / red regressed). Right: the live 2×2 matrix, precision/recall dials, the two overlapping distributions with the threshold, and the cost ledger. The aha: drag base rate down, watch precision collapse — then rescue it with the n slider, not the strictness slider.
Setup. 100 releases/quarter; 20 contain true regressions. Gate A (moderate) detects with probability 0.75, false-blocks good releases with probability 0.10. Gate B (strict) detects with 0.90, false-blocks 0.25. Compute each confusion matrix, precision, recall, and the quarterly cost at $100k per shipped regression and $5k per false block.
Step 1 — Gate A positives. True positives = 20 × 0.75 = 15 regressions blocked. False negatives = 20 − 15 = 5 ship.
Step 2 — Gate A negatives. False positives = 80 × 0.10 = 8 good releases blocked. True negatives = 80 − 8 = 72 ship cleanly.
Step 3 — Gate A metrics. precision = 15 / (15 + 8) = 15/23 = 0.652; recall = 15/20 = 0.75.
Step 4 — Gate A cost. 5 incidents × $100k + 8 false blocks × $5k = $500k + $40k = $540k.
Step 5 — Gate B. TP = 20 × 0.90 = 18, FN = 2; FP = 80 × 0.25 = 20, TN = 60. precision = 18 / (18 + 20) = 18/38 = 0.474; recall = 0.90. Cost = 2 × $100k + 20 × $5k = $200k + $100k = $300k.
Step 6 — the pick depends on the prices. At these costs, strict wins ($300k < $540k) — but note less than half of Gate B's blocks are justified (precision 0.47), a trust problem money doesn't capture. Flip the costs to $20k incidents and Gate A ties: 5 × $20k + $40k = $140k vs 2 × $20k + $100k = $140k — and looser still would win. Neither gate is "right"; the operating point is an economics decision.
Step 7 — base rates dominate intuition. If only 5% of releases truly regress, even a great gate (90% detection, 10% false-block) has:
Two of every three blocks are false — and teams read this as "the gate is broken" when it is just conditional probability. Fixing precision at a low base rate requires shrinking noise (bigger n, more seeds), not moving the threshold.
From scratch — the control-room simulator, ~40 lines, sweeping the threshold for the ROC and cost-optimal point:
python import numpy as np def simulate_gate(n_releases=100, base_rate=0.20, golden_n=2000, true_reg=-2.0, disagree=0.065, seed=0): rng = np.random.default_rng(seed) truth = rng.random(n_releases) < base_rate # True = regressed signal = np.where(truth, true_reg, 0.0) # diff in points noise_sd = 100 * np.sqrt(disagree) / np.sqrt(golden_n) # paired SE in points measured = signal + rng.normal(0, noise_sd, n_releases) return truth, measured def grade(truth, measured, thresh): block = measured < thresh # block if measured drop < thresh tp = np.sum(block & truth); fp = np.sum(block & ~truth) fn = np.sum(~block & truth) prec = tp / (tp + fp) if (tp + fp) else 1.0 rec = tp / (tp + fn) if (tp + fn) else 1.0 cost = fn * 100_000 + fp * 5_000 return prec, rec, int(cost) truth, measured = simulate_gate() for t in [-2, -1, -0.5]: print(t, grade(truth, measured, t)) # stricter t -> higher recall, lower precision
Library equivalent — the gate's PR curve in one call, because the gate is a classifier:
python from sklearn.metrics import precision_recall_curve # y_true = regressed?, scores = -measured_diff (higher = more suspicious) prec, rec, thr = precision_recall_curve(truth, -measured)
The positive class is "regression." Precision = justified blocks / all blocks = 9/30 = 0.30. Recall = regressions caught / all regressions = 9/11 = 0.82. This gate catches most regressions but 70% of its blocks are noise — a classic high-recall, low-precision point that erodes trust; the fix is reducing eval noise (larger n), not just loosening the threshold.
Real teams should audit their gate exactly as this sim does: quarterly, count escaped regressions (from incident reviews) and overturned blocks (from retries and overrides), estimate empirical precision/recall, re-tune. A gate nobody audits drifts into theater (blocks everything, all ignored) or decoration (blocks nothing). Next, Chapter 9 compresses the whole lesson into interview firepower.
Everything in this lesson compresses into one interview superpower: when asked "how do you know your model still works?", you answer with machinery, numbers, and costs — not vibes. Interviewers probing ML reliability are testing whether you distinguish "we have tests" from "we have calibrated gates with known error rates." The single sentence that upgrades any answer: "the gate is itself a classifier, and here is how I'd measure its precision and recall."
These are the calculations an interviewer expects at a whiteboard, no calculator: paired-gate SE = √(disagreements)/n; seeds via (sd/target)²; canary days via the power formula's shape (n scales with 1/δ², so halving the detectable effect quadruples the soak); false-alarm budget = checks × per-check α; and the gate-budget calculation below.
Setup. A team ships 30 releases/month with a 10% true regression rate. Their gate has recall 0.8 and precision 0.7. Compute regressions escaping per month, total blocks per month, and false blocks per month.
Step 1 — true regressions. 30 × 0.10 = 3 per month.
Step 2 — caught (true positives). 3 × 0.8 = 2.4 justified blocks per month.
Step 3 — escapes. 3 − 2.4 = 0.6 regressions ship per month — about 7 incidents a year from this gate alone.
Step 4 — total blocks. Since precision = TP / total blocks, total blocks = 2.4 / 0.7 = 3.43 per month.
Step 5 — false blocks. 3.43 − 2.4 = 1.03 good releases blocked per month — about 12/year of velocity tax.
Step 6 — the follow-up. At $100k/incident and $5k/false block: annual cost = 7.2 × $100k + 12.3 × $5k = $720k + $62k. The escapes dominate, so this team should buy recall (bigger golden set, longer soaks) even at the price of more false blocks.
python def gate_budget(cadence, base_rate, recall, precision, cost_incident=100_000, cost_block=5_000): tr = cadence * base_rate # true regressions tp = tr * recall # caught escapes = tr - tp blocks = tp / precision # total blocks false_b = blocks - tp annual = 12 * (escapes * cost_incident + false_b * cost_block) return round(escapes, 2), round(blocks, 2), round(false_b, 2), int(annual) print(gate_budget(30, 0.10, 0.8, 0.7)) # (0.6, 3.43, 1.03, ...)
Four sliders (data-check catch, offline-gate power, canary power, false-alarm rate) flow 100 releases/quarter through three stacked layers; escapes and false blocks accumulate to a scoreboard. Preset buttons load the trap scenarios — watch "1% canary for 1 hour" leak regressions straight through.
Candidate causes, in order of frequency: golden-set unrepresentativeness (the eval distribution no longer matches live traffic, so offline gains don't transfer), training-serving skew (online features computed differently than offline ones), serving-artifact divergence (quantization/runtime differences mean the shipped binary is not the evaluated checkpoint), and a genuine quality-vs-behavior gap (more "accurate" on the metric but slower, wordier, or different in ways users dislike). Each has a distinct diagnostic: replay live traffic through both models offline (shadow diff), compare online and offline feature values on sampled requests, hash-verify the served artifact against the evaluated one, and decompose the product-metric drop by segment and latency band. Process changes: add a serving-parity check to the gate, refresh the golden set from recent traffic on a schedule, and require a canary phase with product guardrails before full rollout so offline-online divergence is caught at 5% exposure, not 100%.
Follow-up: Which single check would you add first if you could only add one, and why?
First measure the two noise sources: run-to-run eval variance (evaluate one artifact several times; pin temperature, batch size, harness version) and the paired disagreement rate between consecutive weekly models. Gate on the paired diff: count b and c, diff = (c−b)/n, SE = √(b+c)/n, block when the 95% CI lower bound is below −1 point. Check power: with a typical 6–7% disagreement rate, n = 2000 gives SE around 0.57 points, so a 1-point margin is resolvable but marginal — either accept ~80% power, grow toward 4,000, or average multiple eval runs to trim the run-variance component. Add zero-tolerance curated tripwires for past incidents alongside the statistical gate. Log every gate decision with its CI so the quarterly audit can compute the gate's own precision and recall against observed escapes and overturned blocks.
Follow-up: The team complains the gate blocks roughly one good release a month. What do you check before loosening the threshold?
Do not roll back on an input monitor alone — PSI measures that p(x) moved, not that quality fell, and the model may generalize fine over the new region. Investigate in order: identify which bins drive the PSI (per-bin contributions) and whether the shift traces to an upstream pipeline change (a data bug you fix at the source) versus a genuine population change (a campaign, seasonality, a new market); check the output score distribution and slice-level product metrics for the shifted segment specifically, since aggregate flatness can hide a small-segment collapse; and pull a fast labeled sample from the shifted region for a direct quality read. If it's a pipeline bug, fix upstream and consider quarantining affected predictions; if it's a real population shift with stable quality, update the monitoring baseline and consider retraining on the new distribution. Rollback is only on the table if quality evidence — not distribution evidence — shows damage attributable to the current model.
Follow-up: How would your answer change if the shifted feature had the highest importance score in the model?
As stated the gate is impossible: single-run noise exceeds the margin, so any single-run gate is dominated by flakiness. Attack the noise first: pin what is pinnable (temperature 0 if faithful to prod, fixed dataset order, pinned judge version and prompt, fixed batch size or a batch-invariant serving path per the 2025 Thinking Machines work), which removes a large fraction of the wobble. Then average: with residual per-run sd of 1.5 points, k runs give SE = 1.5/√k; k = 9 brings SE to 0.5 points, making a 1-point margin resolvable at ~2 SE. Complement with a larger or better-stratified eval set, since example-sampling noise and run noise add in quadrature — sometimes the cheaper fix is 3× the examples rather than 9× the runs. Finally present the residual tradeoff as a cost curve: margin resolvable versus compute per gate decision, and let the product team pick the point — the margin is a product decision, but the noise floor is physics until paid down.
Follow-up: The eval uses an LLM judge whose provider silently updates the model. What breaks, and how do you detect it?
Split criteria by detection speed and blast radius. Fast automatic tier (minutes, no human): serving health — error rate, timeout rate, score-distribution collapse (everything suddenly 0.99, or NaN rates) — each with Poisson/z tolerances from historical noise, because these indicate breakage, not judgment calls. Medium tier (hours, page a human): decline-rate and score-distribution shifts on the canary arm versus control beyond pre-registered bands — decline rate is observable immediately even though fraud labels are not. Slow tier (days–weeks): actual fraud outcomes (chargebacks) are lagged, so the canary cannot gate on true fraud recall directly; pre-register a post-hoc review at label maturity with an agreed reversal criterion. The asymmetric costs shape the bands: the decline-rate band is asymmetric, tighter against declines falling (which may mean fraud passing). Pre-register everything before the canary starts, test the rollback path mechanically beforehand, and randomize by account, not transaction.
Follow-up: Fraud labels take three weeks to mature. Does that change how long you keep the old model warm?
Two independent flaws. Power: 1% of even a large service for one hour yields a sample that can only detect enormous effects — with 200k daily requests, one hour of 1% is roughly 83 users; against a conversion-style metric the detectable effect at that n is tens of relative percent, so any subtle regression sails through and the canary is theater. The math: n per arm scales as 1/δ², so halving the detectable effect quadruples the required sample. Exposure: canaries put real users in front of the regression — the offline gate exists precisely because it catches the cheap majority with zero user exposure and minutes of compute; removing it converts every catchable offline failure into user-facing exposure plus a slow, underpowered online detection problem. The correct architecture keeps both: offline gates filter cheaply and fast, canaries catch what offline structurally cannot. Concede what's right: live traffic IS the ground truth, which is why the canary is the last gate — not the only one.
Follow-up: What is the minimum canary size and duration you'd defend for their service, and what numbers do you need to compute it?
Treat the prompt as a release artifact with the full gate stack, because prompt is a first-class regression surface. Offline: run the paired golden-set gate — same examples through old and new prompts, count disagreements, CI on the diff — plus targeted suites for the behaviors prompts commonly break: structured-output parse rate, refusal rate, tool-call validity, output length (a cost gate — verbose prompts silently raise serving cost), and the curated tripwire set from past prompt incidents. Account for nondeterminism with k-run averaging since prompt diffs are often small relative to sampling noise. Then shadow the new prompt on mirrored traffic to measure the live flip rate and diff changed responses by slice; a "improvement" that flips 25% of production responses needs scrutiny far beyond its golden-set delta. Finally canary with parse-failure and length monitors. Cultural point: version prompts in the repo, require the same review-and-gate pipeline as model binaries, and hash the deployed prompt so serving parity is verifiable — the GPT-4o sycophancy episode came from a behavioral change that looked innocuous in spot checks.
Follow-up: Your prompt eval uses an LLM judge. The new prompt makes outputs longer and the judge favors longer answers. How do you keep the gate honest?
Compute the gate's own metrics first: precision 6/28 = 21%, recall 6/9 = 67%. Both poor, and the combination — many false blocks AND several escapes — is the signature of high measurement noise rather than a mis-set threshold, because moving the threshold alone only trades one error for the other. Check the base rate: 9 regressions in ~100+ releases is under 10%, so some precision pressure is structural, but 21% indicates the noise distributions overlap heavily. Plan: (1) measure run-to-run sd and the golden set's power for the enforced margin — likely undersized; (2) grow or re-stratify the golden set and adopt k-run averaging to shrink measured-diff noise, improving precision and recall simultaneously; (3) re-derive the threshold from the incident-to-block cost ratio; (4) add layer diversity if escapes cluster in one surface (e.g., all three escapes were data regressions — then the fix is validation/drift coverage, not the offline gate at all). Numeric targets: with noise halved, expect precision above 50% and recall above 85% at the same cost-optimal threshold; re-audit next quarter.
Follow-up: The three escapes were all latency regressions. What does that tell you about where the fix belongs?
Legitimate overrides exist — gates encode a policy and policies have edge cases: a security fix whose urgency dominates a marginal quality risk, a gate failure traced to a broken eval (judge outage, corrupted golden partition) rather than the candidate, or a known-acceptable tradeoff the margin doesn't encode (deliberately trading 0.5 points of aggregate accuracy for a large fairness improvement on a protected slice). A healthy process makes these auditable, not convenient: overrides require a named approver distinct from the release author, a written rationale on the release record, an automatic follow-up task to validate the risk post-hoc, and a hard budget — override rate is itself a monitored metric, because a rising rate means either the gate is miscalibrated (fix the gate) or discipline is eroding (fix the culture). What must never happen: silent retries-until-green as de facto overrides, or "emergency" becoming a routine label. In the quarterly audit, overridden-then-fine releases count as gate false positives, overridden-then-incident as self-inflicted escapes — both feed recalibration.
Follow-up: Your override rate doubled after a reorg. What hypotheses do you form, and what data distinguishes them?
The +0.5 is a point estimate; without n (and the disagreement count) it is uninterpretable — on n = 500 the CI spans multiple points both ways. Asking for the sample size and reasoning about the interval demonstrates exactly the calibrated-gate thinking the question probes. (Then, and only then, do canary considerations enter.) Chapter 10 ties every layer into one map — and one final calculation.
One last calculation ties the whole lesson together: what is the probability a regression makes it past everything? The answer reveals why layering works — and the one condition that makes it a lie.
If a data regression slips past data validation half the time, past the offline golden-set gate a quarter of the time, and past the canary 30% of the time, the full-stack escape probability is the product: 0.5 × 0.25 × 0.30 = 3.75% — versus 25–50% for any single layer. Defense in depth is not redundancy theater; it is multiplication. Play with the map: toggle layers and watch the escape percentage recompute; flip the "correlated" switch and watch the multiplication break.
This lesson at center; sibling and adjacent lessons orbit it, edges labeled with what flows between them. Tap a node to read its ownership blurb. The bottom strip runs the layered-escape calculation live — toggle each defense layer and watch the escape % recompute; the correlated toggle demonstrates the independence caveat by inflating the product.
Setup. A data regression must pass three independent layers: validation misses it 0.5, the offline gate misses it 0.25, the canary misses it 0.30. (a) Full-stack escape? (b) If the golden set and canary are correlated so a regression missed offline is missed by the canary 60% of the time (not 30%), the true escape rate?
Step 1 (a) — independent layers multiply. P(escape) = 0.5 × 0.25 × 0.30.
Step 2. 0.5 × 0.25 = 0.125. 0.125 × 0.30 = 0.0375 — a 3.75% escape rate, versus 25–50% for any single layer alone.
Step 3 — the funnel. Per 100 true data regressions: 50 pass validation, 12.5 pass the offline gate, 3.75 reach users.
Step 4 (b) — with correlation. P(escape) = 0.5 × 0.25 × 0.60 = 0.075 — 7.5%, double the independent estimate. Correlated layers quietly halve your protection while the architecture diagram looks identical.
Step 5 — the design lesson. Independence is a property you must engineer (different metrics, different traffic, different failure modes per layer), not one you get for free by adding boxes.
python import numpy as np print(np.prod([0.5, 0.25, 0.30])) # 0.0375 — IF independent print(np.prod([0.5, 0.25, 0.60])) # 0.075 — correlated canary
This lesson owns the release-gate and monitoring machinery. Each layer's deeper theory is its own lesson:
| Lesson | What it owns / why it connects |
|---|---|
| Metric Design | Prerequisite. The gate can only be as good as the metric it enforces — metric definitions and design theory live there. |
| The Statistics of Evaluation | Every CI, SE, power formula, and multiple-testing correction our gates use is derived there; this lesson only operationalizes them. |
| Experiment Design & A/B Testing | Canaries are safety-scoped A/B tests; the full causal machinery, sequential testing, and CUPED variance reduction live there. |
| Plots That Tell the Truth | The diff dashboards, control bands, and drift charts our monitors render honestly are its subject. |
| Evaluating Generative Models | pass@k, judge-bias quantification, and generative metrics that plug into these gates for LLM/VLM/diffusion. |
| Evaluating Estimators, Fusion & Robots | NEES/NIS-style estimator consistency checks are the robotics instance of a regression gate. |
| The Metrics Ladder | Translating gate precision/recall and incident costs into product and exec language. |
| AI Evaluation | Its LLM regression suites are the process-level view; this lesson supplies the statistical machinery under them. |
| Testing & Deployment | Blue/green, feature flags, CI/CD plumbing — how releases physically move; our gates bolt onto that infrastructure. |
| Evaluation (CS336) | Benchmark contamination and perplexity mechanics — why golden-set leakage hygiene (Ch2) matters. |
The independence caveat that keeps the multiplication honest: layers only multiply if their failure modes are independent. If your golden set and your canary guardrail measure the same metric on similar traffic, they share blind spots, and the true escape rate is higher than the product. Diversify what each layer looks at: inputs (validation), distributions (drift), quality (golden sets), and live behavior (canary).
So you know where to go next: multiple-comparison corrections and sequential testing (eval-statistics), CUPED and variance reduction for faster canaries (experiment-design), LLM-judge drift as an eval-infrastructure regression (genai-eval), and org-level quality metrics (metrics-ladder).
| Level | What you have |
|---|---|
| 0 | Tests on code only |
| 1 | Golden set exists, compared by eyeball |
| 2 | Statistical gates with margins and measured run variance |
| 3 | Data validation + drift monitoring in production |
| 4 | Canary with pre-registered auto-rollback |
| 5 | The gate itself is audited with its own precision/recall |
Most teams believe they are at 3 and are actually at 1.
| Tool | Formula | When |
|---|---|---|
| Paired gate | diff = (c−b)/n, SE = √(b+c)/n; block if lo < −δ | Offline release decision on a golden set |
| Golden-set size | n = (zα+zβ)² × pdisagree / δ² | Sizing a set to police a margin δ |
| Seeds needed | k = (sd / SEtarget)² | Averaging out run-to-run noise |
| PSI | Σ (a−e) × ln(a/e) | Binned tabular feature drift (magnitude) |
| KS statistic | D = max |CDFa − CDFe|; gate on D at scale | Continuous feature drift (detectability) |
| Null-rate check | z = (rate−p) / √(p(1−p)/n) | Data validation with a noise-scaled band |
| Canary power | n = (zα+zβ)²(p1q1+p2q2) / Δ² | How long a canary must soak |
| Percentile | nearest-rank: sorted[⌈p/100 × n⌉] | Latency gates (pin the definition!) |
| Fan-out tail | 1 − (1−q)k | Tail amplification across k backends |
| Gate metrics | precision = TP/(TP+FP), recall = TP/(TP+FN) | Grading the gate itself |
| Stack escape | ∏ missi (only if independent) | Defense-in-depth escape probability |
Multiplication of miss rates requires independent failure modes. Team A diversified what each layer observes, so a regression invisible to one can still trip another. Team B re-tests the same signal three times: any regression outside that metric's view escapes all three together, making the 3.75% figure a fiction. Engineer independence; don't assume it.
Every chapter reduced to the same move — treat a noisy measurement as a distribution, bound the downside, pre-commit the decision rule, and audit the decider. That move is the entire discipline of evaluation engineering.