Your feature "improved" retention by 5 points — because it launched the week before a holiday. This lesson teaches you to measure what YOU caused, not what the calendar did: randomization, power, peeking, CUPED, switchbacks, and why offline wins vanish online.
Your team ships a new "For You" recommendations panel in a music-streaming app on December 18th. You watch the weekly-retention chart the next Monday: it climbed from 40% to 45%. Five points. The launch email practically writes itself — "For You panel drives +5pp retention." Champagne.
Then a quiet skeptic on the team pulls up last year's chart for the same week. Retention always jumps about 5 points that week. It jumped last year with no launch at all. It's the holidays: people are home, bored, and streaming. Your +5pp "win" was the calendar, not your code.
The method that fooled you has a name: a before/after comparison (sometimes dressed up as "pre-post analysis") — you measure the metric before the change, measure it after, and attribute the difference to the change. It feels like the obvious thing to do. It is also one of the most reliable ways to ship a fantasy.
Here is the villain by its proper name. A confounder is a variable — here, holiday free time — that changed at the same time as your treatment and independently moves the metric. The before/after comparison cannot tell your feature's effect apart from the holiday's effect, because the two are perfectly entangled: everyone who got the "after" also got the holiday.
Watch the arithmetic manufacture a phantom. Ten thousand users the week before launch retain at 40% — that's 4,000 people. Ten thousand different users the launch week retain at 45% — 4,500 people. Feed that into the two-proportion z-test from eval-statistics: you get z ≈ 7.15, p ≈ 8.6×10-13. The test screams significance. And it is completely right: the two weeks genuinely differ. It just cannot tell you why. Significance is not causation.
The fix fits in one sentence. Split users randomly at the same moment in time: flip a coin per user, give half the feature (treatment) and half the old experience (control). Now both arms live through the exact same holiday — the holiday lifts both equally, so it cancels when you subtract control from treatment. Randomization is a confounder-eraser: it balances everything you thought of and everything you didn't.
Run the honest version. Randomize 50/50 during launch week. The holiday lifts everyone, so both arms retain around 45%. Treatment minus control ≈ 0. The feature did nothing. That is a painful Monday — but it is a much cheaper lesson than scaling a dud to a hundred million users and wondering next quarter why retention is flat.
Four weeks of daily retention for a streaming app, with a holiday bump baked into the world. A launch marker splits the data. Toggle how you assign users: by time (before launch = control, after = treatment) vs randomly (each user coin-flipped, both arms span all weeks). Set the feature's true effect — then watch time-assignment always add the holiday's phantom, while random assignment recovers the truth.
Let us make the phantom concrete with the full arithmetic, so you can reproduce every digit. Same world: the feature's true effect is exactly zero; the holiday adds +5pp to everyone.
1. Naive time comparison. Control (pre-week) retains 4,000/10,000 = 0.40. "Treatment" (launch week) retains 4,500/10,000 = 0.45. Observed lift = 0.45 − 0.40 = 0.05 (+5pp).
2. Pooled proportion. p̂ = (4000 + 4500) / (10000 + 10000) = 8500/20000 = 0.425.
3. Standard error. SE = √( 0.425 × 0.575 × (1/10000 + 1/10000) ) = √(0.244375 × 0.0002) = √(0.000048875) = 0.006991.
4. z-statistic. z = 0.05 / 0.006991 = 7.152. Two-sided p-value = 8.6×10-13. Wildly "significant" — and 100% confounded, because the true feature effect is zero by construction.
5. Randomized version. Both arms live in launch week, so both get the holiday's +5pp. Control retains at 45%; treatment retains at 45% + true effect 0 = 45%. Observed lift = 0.45 − 0.45 = 0. The holiday appears identically in both arms and cancels in the subtraction.
Moral in numbers: the same statistical test gave z = 7.15 for a nonexistent effect under time assignment, and correctly ~0 under randomization. The test was never broken — the design was.
Now the same computation in code — first from scratch, so nothing hides in a library call, then the one-liner that gives the identical answer.
python — from scratch import numpy as np from scipy import stats # Naive time-based comparison: pre-week vs launch week c_conv, c_n = 4000, 10000 # control = pre-launch week t_conv, t_n = 4500, 10000 # "treatment" = launch (holiday) week p_pool = (c_conv + t_conv) / (c_n + t_n) se = np.sqrt(p_pool * (1 - p_pool) * (1/c_n + 1/t_n)) lift = t_conv/t_n - c_conv/c_n z = lift / se p_val = stats.norm.sf(z) * 2 print(round(p_pool,3), round(se,6), round(z,3), p_val) # -> 0.425 0.006991 7.152 8.55e-13 ... a phantom "discovery" # Randomized version: both arms in launch week, feature effect = 0 rng = np.random.default_rng(0) n = 20000 holiday = 0.05 # +5pp to everyone this week treat = rng.random(n) < 0.5 # fair coin per user base = 0.40 + holiday # 0.45 baseline this week retain = rng.random(n) < (base + 0.0 * treat) # true effect 0.0 print(retain[treat].mean() - retain[~treat].mean()) # -> ~0.00
python — library equivalent from statsmodels.stats.proportion import proportions_ztest z, p = proportions_ztest([4000, 4500], [10000, 10000]) print(round(z,3), p) # -> -7.152 8.6e-13 (same magnitude — same answer)
Widen the lens, because this failure mode is everywhere in ML products. "The new model reduced fraud" — and the fraud ring got arrested that month. "The new prompt improved CSAT" — and support hired more staff. "The new ranker lifted clicks" — and paid-traffic mix shifted. Evaluation without a control group is astrology with confidence intervals.
Immediately name confounding: everything else that changed that month (seasonality, marketing, mix shift, other launches) is entangled with the model. The p-value is real but answers the wrong question — "do the periods differ," not "did the model cause it." Propose (a) a proper randomized holdback going forward, even 5–10% of traffic; (b) if that's impossible retroactively, quasi-experimental fallbacks in order of credibility — difference-in-differences against an unaffected segment/region, interrupted time series with last year's seasonal baseline, or synthetic control. A staff-level candidate also quantifies the cost of being wrong (scaling a dud, opportunity cost) to justify holding the announcement.
So the whole game is design, not test statistics. But why, exactly, does a coin flip rescue a comparison that arithmetic alone cannot? To answer that we need the language of ghosts and counterfactuals — the question an A/B test is actually answering. That's Chapter 1.
For every user who got your feature, there is a ghost: the same user, same day, same mood — without the feature. The causal effect on that user is you minus your ghost. The tragedy of causal inference is that the ghost is never observed.
Let us make that precise. Each user u carries two sealed envelopes. Call them potential outcomes: Yu(1) is their outcome if treated, and Yu(0) is their outcome if not treated. For our streaming app, each is 1 if the user retains next week, 0 if they churn.
The individual treatment effect is the difference between the two envelopes: Yu(1) − Yu(0). But opening one envelope burns the other — a user either got the feature or didn't; you can never see both outcomes for the same person. This is the fundamental problem of causal inference, and no amount of data on one user fixes it.
So we lower our sights. We give up on individual effects and target the average treatment effect (ATE) — the average, across everyone, of their envelope-differences. This is estimable precisely because averages let different users stand in for each other's ghosts.
Six podcast-app users, each holding two envelopes: Y(0) grey (churn/retain without the feature) and Y(1) purple (with it). God view reveals both and the true ATE. In mortal view you open exactly one per user: Self-selection (engaged users grab treatment — the observed gap inflates to 1.0) vs Coin flip (repeated random assignments; the running average converges onto the true ATE line).
Now walk the table like an omniscient god who can peek in every envelope. Six users; the god-view potential outcomes are Y(0) = [1,1,0,0,1,0] and Y(1) = [1,1,1,0,1,1].
1. Individual effects Y(1)−Y(0) per user: 1−1=0, 1−1=0, 1−0=1, 0−0=0, 1−1=0, 1−0=1.
2. True ATE = (0+0+1+0+0+1)/6 = 2/6 = 0.3333. The feature genuinely helps two of the six users.
3. Self-selection. Let the three already-engaged users — users 1, 2, 5, exactly those with Y(0)=1 — opt into treatment. We observe their Y(1) = [1,1,1], mean = 3/3 = 1.0. Users 3, 4, 6 stay control; we observe their Y(0) = [0,0,0], mean = 0/3 = 0.0.
4. Naive observed difference = 1.0 − 0.0 = 1.0 — three times the true ATE of 0.333.
5. Decompose. Selection bias = observed difference − true ATE = 1.0 − 0.3333 = 0.6667. Source: the treated users' ghosts Y(0) = [1,1,1] (they'd have retained anyway) versus the controls' observed Y(0) = [0,0,0]. The groups were never comparable.
6. Randomization check. Under a fair coin every user is equally likely to land in either arm, so the expected treated-minus-control difference equals the ATE = 0.3333. Any single flip is noisy with n=6 (that's Chapter 3's problem), but the bias term is exactly zero in expectation.
The punchline is a decomposition worth tattooing on your wrist. In words:
Correlation ("treated users retain more") always decomposes into causation plus "treated users were different to begin with." Every observational ML claim — "users of the new editor churn less," "sessions with the AI assistant convert more" — carries an unknown selection term you can't see.
Now the mechanical heart of the lesson: why does randomization zero out that selection term? A coin flip is independent of both envelopes. So the treated sample's Y(0) ghosts have the same distribution as the control group's observed Y(0). The control average is therefore an unbiased estimate of the treated group's missing counterfactual, and the selection-bias term is zero in expectation — for observed and unobserved traits alike.
That last clause is the superpower. No matching or modeling on covariates can promise balance on things you never measured — motivation, life events, taste. A coin flip balances them all, because it doesn't need to know they exist.
Be honest about the fine print. Randomization gives unbiasedness in expectation, not exact balance in any one experiment — small samples can be unlucky (Chapter 3's power, Chapter 5's covariate adjustment). And it needs no interference between users: one user's treatment must not touch another's outcome (Chapter 6). Both caveats are real; neither undoes the guarantee.
Here is the code. There is no library shortcut for the counterfactual itself — that is the whole point — but the RCT estimator is just treated.mean() - control.mean().
python — from scratch import numpy as np Y0 = np.array([1,1,0,0,1,0]) # outcome WITHOUT feature Y1 = np.array([1,1,1,0,1,1]) # outcome WITH feature ite = Y1 - Y0 ate = ite.mean() print("ITE", ite, "ATE", round(ate,4)) # -> [0 0 1 0 0 1] 0.3333 # Self-selection: engaged users (Y0==1) grab treatment treated, control = [0,1,4], [2,3,5] obs = Y1[treated].mean() - Y0[control].mean() print("self-select diff", obs, "bias", round(obs-ate,4)) # -> 1.0 0.6667 # Random assignment converges to the ATE over many flips rng = np.random.default_rng(0) diffs = [] for _ in range(10000): t = rng.random(6) < 0.5 if t.any() and (~t).any(): diffs.append(np.where(t, Y1, Y0)[t].mean() - np.where(t, Y1, Y0)[~t].mean()) print("mean random diff", round(np.mean(diffs),3)) # -> ~0.333
One more vocabulary bridge: "A/B test" = randomized controlled trial = the only tool in this lesson that estimates the counterfactual by construction rather than by assumption. Everything else you'll meet — diff-in-diff, synthetic control, propensity matching — buys causality with assumptions you must defend (Chapter 10).
The 30% is observed difference = ATE + selection bias: enablers are power users whose ghosts (retention without the feature) already exceed non-enablers'. Auto-enabling delivers the treatment to the marginal population, whose effect may be near zero or negative. The honest estimate requires randomizing auto-enablement among current non-enablers (or an encouragement design if forcing is unacceptable) and measuring the effect on that population. Bonus: name the estimand shift — effect-on-the-treated vs effect-on-the-untreated — and note that even the direction can flip: features loved by opt-in enthusiasts often annoy everyone else.
Randomization is the whole ballgame — but "flip a coin per user" hides three separate engineering decisions, and getting any of them wrong silently corrupts the trial. What do you flip the coin on? How does the coin stay consistent? How do you prove it wasn't rigged? That's Chapter 2.
Randomization sounds like one decision. It is three: what you flip the coin on, how the coin stays consistent, and how you prove the coin wasn't rigged. Botch any one and the cleanest statistics in the world report on a corrupted trial.
Decision 1 — the randomization unit. Flip the coin per user: each person gets one consistent experience. Flip per session or per request: the same person sees both arms across visits. The rule of thumb: randomize at the level where the treatment is experienced and the metric is defined. A UI change users can notice must be per-user — showing someone both versions is its own weird treatment. A backend latency tweak nobody perceives can be per-request.
Here is the subtle part: the unit changes the metric itself. Take three users' sessions — user A: 4 sessions, 2 convert; user B: 1 session, 1 converts; user C: 5 sessions, 0 convert. Session-level conversion is 3/10 = 30%. User-level conversion (average of per-user rates) is (0.5 + 1.0 + 0.0)/3 = 50%. Same raw log, two different numbers. Neither is wrong — they answer different questions — but mixing them (randomize by session, report per-user) is incoherent.
There's a statistical trap hiding in the unit choice too. Your statistics from eval-statistics assume independent observations. Sessions from the same user are correlated — one heavy user contributes 50 correlated sessions. If you randomize per-user but compute session-level standard errors as if independent, your confidence intervals come out too narrow and false positives spike. The fix: analyze at the unit of randomization (per-user means), or use the delta method / clustered SEs for ratio metrics.
Decision 2 — assignment must be deterministic and stateless. The pattern is hash(user_id + experiment_salt) → bucket 0–99; the first 50 buckets are control. The same user always lands in the same arm across devices and days — with no assignment database to consult. The per-experiment salt decorrelates concurrent experiments: without it, the same users land in "treatment" for every experiment at once and experiments contaminate each other.
Also beware carryover: reusing a salt (or bucketing) from a previous experiment means your "random" groups inherit the previous treatment's leftover effects. Platforms re-salt or shuffle buckets between experiments for exactly this reason.
Decision 3 — trust, but verify: the Sample Ratio Mismatch (SRM) check. You planned 50/50; you observe 50,600 vs 49,400. Your eyeball says "fine." A chi-square test on the assignment counts says the coin was rigged: χ² = 14.4, p = 0.00015. Under a fair 50/50 split of 100k units, a 600-user imbalance is a five-in-ten-thousand event.
User-id chips flow through a hash into 100 buckets (first 50 teal = control, last 50 purple = treatment). Change the salt and the same users reshuffle. Now the danger: crank event loss to silently drop a fraction of ONE arm's users (simulating client crashes) — the split still looks even, but the SRM dial swings into the red. Run the check to stamp PASS/FAIL with the live χ² and p.
Now the two computations by hand — the unit-changes-the-metric arithmetic, then the SRM chi-square.
Session-level conversion = total conversions / total sessions = (2+1+0)/(4+1+5) = 3/10 = 0.30.
User-level conversion (average of per-user rates): A = 2/4 = 0.5, B = 1/1 = 1.0, C = 0/5 = 0.0; mean = (0.5+1.0+0.0)/3 = 1.5/3 = 0.50.
Same log, 30% vs 50% — heavy user C drags the session metric down but counts once in the user metric. The randomization unit must match the metric's denominator.
1. Expected counts under 50/50: E = 100,000/2 = 50,000 per arm.
2. Chi-square = (50600−50000)²/50000 + (49400−50000)²/50000 = 600²/50000 + 600²/50000 = 360000/50000 × 2 = 7.2 + 7.2 = 14.4.
3. Degrees of freedom = arms − 1 = 1. p-value = P(χ²1 ≥ 14.4) = 0.000148.
Verdict: a 1.2% relative imbalance on 100k units is a p = 0.00015 event — SRM FAIL. Do not read the metrics; find the differential loss (segment the SRM check by platform/country/browser to localize it).
Why is SRM a red alert and not a shrug? The imbalance is almost never the coin — it's differential loss after assignment. Classic causes: the treatment crashes old Android clients (their events never arrive), bot filters trip on one arm, redirect links drop users, logging changed in one arm. The users you lost are systematically different, so every downstream metric is biased. The experiment result must be discarded, not "adjusted." Microsoft's ExP team found roughly 6% of experiments trip SRM — this is a daily alarm, not a theoretical one.
python — from scratch import hashlib, numpy as np from scipy import stats def assign(user_id, salt): h = int(hashlib.md5(f"{user_id}:{salt}".encode()).hexdigest(), 16) bucket = h % 100 return "control" if bucket < 50 else "treatment" arms = [assign(uid, "exp_42") for uid in range(100000)] c = arms.count("control"); t = arms.count("treatment") # chi-square by hand on the counts we planned to be 50/50 E = (c + t) / 2 chi2 = (c - E)**2/E + (t - E)**2/E print(c, t, round(chi2,2), stats.chi2.sf(chi2, 1)) # now DROP 2.4% of treatment events (client crash) and re-check t_lost = int(t * 0.024); t2 = t - t_lost E2 = (c + t2) / 2 chi2_bad = (c - E2)**2/E2 + (t2 - E2)**2/E2 print("after loss:", c, t2, round(chi2_bad,1), stats.chi2.sf(chi2_bad, 1)) # SRM FAIL
python — library equivalent from scipy.stats import chisquare chi2, p = chisquare([50600, 49400], f_exp=[50000, 50000]) print(round(chi2,1), round(p,7)) # -> 14.4 0.0001478 (same answer)
Make the check a habit: run the chi-square on assignment counts — and per segment (platform, country, new/returning) — automatically, before reading any metric. An experiment result without an SRM check is an unverified experiment.
Core guarantees: deterministic and sticky (hash of unit id + per-experiment salt, no assignment DB), independent across experiments (unique salts so bucket membership decorrelates), consistent across devices (choose a durable id — logged-in user id beats device id), and re-salted between iterations to kill carryover. Layered / mutually-exclusive domains for experiments that interact. Detection: automated SRM chi-square on every experiment overall and per segment, plus an A/A pipeline continuously validating the whole stack end-to-end. Strong candidates mention that ~6% of experiments at Microsoft trip SRM — a daily alarm — and that an SRM failure invalidates the experiment rather than being correctable in analysis.
Suppose your assignment layer is flawless and SRM passes. You still can't run the test until you answer the most-asked question in all of experimentation: how many users, and for how long? That number falls out of a formula — and hides a trap that has burned entire quarters. Chapter 3.
The most common experiment failure isn't a false positive. It's the test that never had a chance — two weeks of traffic chasing an effect that needed six months to see. A "not significant" result that was doomed from the start.
Reframe the two error types from eval-statistics in shipping terms. Alpha (false positive rate, typically 0.05) is the probability you ship a dud believing it works. Beta is the probability you miss a real effect. Power = 1 − beta (typically 0.80) is the probability you detect the effect if it is really there. An underpowered experiment mostly produces "not significant" regardless of the truth — it's a coin you paid two weeks of traffic to not flip.
Define the minimum detectable effect (MDE): the smallest true lift you'd care to detect. It is not a prediction — it's a business decision. "If it's smaller than X, it isn't worth shipping the complexity for." Everything downstream is a negotiation between MDE, traffic, and patience.
This is the same power/sample-size machinery derived in eval-statistics; we don't re-derive it, we re-read it for the online, two-arm setting. Recall the formula from two constraints. Constraint 1 (alpha): under no effect the observed difference is noise with standard deviation SE; we only call significance when the difference exceeds zα/2·SE. Constraint 2 (power): under a true effect δ, the observed difference is centered at δ; for 80% of experiments to clear the significance bar, δ must sit zβ SEs above it. Stack them:
What is new here is the online substitution — SE for two proportions. Solving for n per arm:
Read that like an engineer. n scales with variance (noisier metrics need more users) and with 1/δ² — the brutal inverse-square law of experimentation. Detecting an effect half as big costs 4× the sample; a tenth as big costs 100×.
Two sampling distributions of the observed lift: null centered at 0, alternative centered at the MDE, with the significance threshold as a vertical line. The red tail past the line under the null is alpha; the green area past it under the alternative is power. Drag any slider and watch the curves morph and the live readout — n per arm and days-to-run — update. Flip the relative vs absolute toggle to feel the 367× trap.
Now the worked example, every step shown. Checkout experiment: baseline conversion p1 = 5%, detect +1 absolute point (p2 = 6%), two-sided α = 0.05, power = 80%. Daily eligible traffic 20,000 users, 50/50 split.
1. Critical values. zα/2 = z0.975 = 1.960 (from eval-statistics), zβ = z0.80 = 0.842. Sum = 2.802, squared = (2.802)² = 7.849.
2. Variance terms. p1(1−p1) = 0.05 × 0.95 = 0.0475; p2(1−p2) = 0.06 × 0.94 = 0.0564. Sum = 0.1039.
3. Effect. δ = 0.06 − 0.05 = 0.01, δ² = 0.0001.
4. n per arm. 7.849 × 0.1039 / 0.0001 = 0.81550 / 0.0001 = 8,155 users per arm (8154.99 rounded up), total = 16,310.
5. Duration. 16,310 / 20,000 per day = 0.82 days to fill — but run 7–14 days anyway for weekly cycles.
6. The relative trap. "+1% relative": p2 = 0.05 × 1.01 = 0.0505, δ = 0.0005, δ² = 2.5×10-7. Variance sum = 0.0475 + 0.0505×0.9495 = 0.095450. n per arm = 7.849 × 0.095450 / 2.5e-7 = 2,996,694.
7. Ratio. 2,996,694 / 8,155 = 367× more users for the ambiguous "other" reading of "1% lift." Duration: 5,993,388 / 20,000 = 300 days. Absolute vs relative MDE is a 1-day vs 10-month difference — write the units down.
That is the trap the chapter title promised. "1% lift" is fatally ambiguous. One percentage point (5% → 6%) needs 8,155/arm. One relative percent (5% → 5.05%) needs nearly three million per arm. Teams have burned quarters on this ambiguity; always write MDE in absolute points alongside the relative figure.
python — from scratch import numpy as np from scipy import stats def sample_size(p1, p2, alpha=0.05, power=0.80): za = stats.norm.ppf(1 - alpha/2) # 1.960 zb = stats.norm.ppf(power) # 0.842 var = p1*(1-p1) + p2*(1-p2) delta = p2 - p1 n = (za + zb)**2 * var / delta**2 return np.ceil(n) for p2 in (0.06, 0.0505): n = sample_size(0.05, p2) print(p2, int(n), "per arm ->", round(2*n/20000, 2), "days") # -> 0.06 8155 per arm -> 0.82 days # -> 0.0505 2996695 per arm -> 299.7 days
python — library equivalent from statsmodels.stats.power import NormalIndPower from statsmodels.stats.proportion import proportion_effectsize es = proportion_effectsize(0.06, 0.05) n = NormalIndPower().solve_power(effect_size=es, alpha=0.05, power=0.8) print(round(n)) # -> ~8000 (arcsine effect-size transform; same ballpark)
Duration math with real numbers: with 20,000 eligible users/day split 50/50, the 1pp test fills in 16,310/20,000 = 0.82 days — but you still run 1–2 full weeks to cover day-of-week cycles and novelty (Chapter 6). The 1%-relative test needs 5,993,388/20,000 = 300 days: the honest answer is "we cannot run that test." Accept a coarser MDE, find a more sensitive metric, or reduce variance (Chapter 5).
(1) Reduce variance instead of adding sample: CUPED with a pre-period covariate (Chapter 5) — a 0.7 correlation cuts required n by half; stratification and winsorizing heavy tails also help. (2) Move to a more sensitive surrogate metric with higher signal-to-noise per user (defer the metric choice to metric-design / metrics-ladder). (3) Change the design: within-subject / paired or interleaving designs (Chapter 6) can be orders of magnitude more sensitive for ranking changes; sequential tests spend traffic more efficiently across many experiments. A staff answer includes the honest fourth option: accept the 3pp MDE and make the ship/no-ship framework explicit about undetectably-small effects.
You've computed a horizon: "run 14 days, then read the result." But no one waits. Day 3 the dashboard flashes p = 0.03, and the temptation to stop is overwhelming. That impatience quietly manufactures false winners — and it's the subject of Chapter 4.
Day 3 of your two-week test, the dashboard shows p = 0.03. Ship it? Here's the uncomfortable fact: the dashboard will show p < 0.05 at some point in most A/A tests too — tests where there is no real effect at all — if you look often enough.
Set the scene honestly: every experimenter peeks. Dashboards update hourly; the pull to stop early on a green number is structural, not a character flaw. The real question is what peeking does to your error rates.
The mechanism is a drunk walk. Under the null, the z-statistic wanders randomly as data accrues. A fixed-horizon test asks one question: "is |z| > 1.96 at the end?" — a 5% chance. Peeking asks a different question: "does |z| ever exceed 1.96 at any of my 20 looks?" The maximum of a random walk crosses a fixed line far more often than its endpoint does.
An A/A test (true effect zero): the z-statistic random-walks over 20 days between the ±1.96 rails. If it ever crosses, the trace flashes red — a false winner. Three policies race: Fixed-horizon (check day 20 only), Naive peeking (stop at first crossing), and O'Brien-Fleming (a funnel that starts wide ~4.5σ and narrows to ~2). Run 500 and watch the meters settle near 5%, 25%, and 5%.
Now quantify the damage exactly with a simulation you can reproduce. 20,000 A/A experiments (true effect = 0). Each runs 20 days at 500 users/arm/day. Policy A: test once at day 20 with |z| > 1.96. Policy B: test every day and declare a winner at the FIRST |z| > 1.96.
1. Per-day data. Each arm draws 500 users/day from a unit-variance null; after d days an arm has nd = 500d users and its mean has variance 1/nd.
2. Daily z. zd = (meanT − meanC) / √(2/nd). Each zd is standard normal — but the sequence (z1…z20) is heavily autocorrelated, because later means reuse earlier data.
3. Policy A (fixed horizon). Fraction with |z20| > 1.96 = 0.0492 — right at the promised 5%.
4. Policy B (daily peeking). Fraction where any of the 20 looks has |zd| > 1.96 = 0.2539 — a 25.4% false-positive rate. Same data, same statistic, 5× the error, purely from the stopping rule.
5. Why not 20×5% = 100%? The looks are positively correlated (each day shares most of its data with the previous day), so crossings cluster; the union is far above 5% but far below the Bonferroni-style sum.
6. Boundary check. Bonferroni would demand per-look p < 0.05/20 = 0.0025 (|z| > 3.02) — valid but conservative. O'Brien-Fleming spends almost nothing early (day-1 boundary near |z| = 8.8 for 20 looks; ~4.5 for 5 looks) and ends near 2.04, preserving ~98% of fixed-horizon power while capping total alpha at 5%.
Your alpha didn't drift from 5% to 6%. It quintupled. And it keeps climbing with more looks: continuous monitoring pushes it toward certainty (the law of the iterated logarithm guarantees z eventually crosses any fixed bound). Peeking-with-early-stopping also biases the effect estimates of "winners" upward — you stopped precisely when noise peaked (the winner's curse, Chapter 10).
python — from scratch (seeded, reproducible) import numpy as np rng = np.random.default_rng(0) S, D, n = 20000, 20, 500 # sims, days, users/arm/day t = rng.normal(0, np.sqrt(n), (S, D)) # daily SUMS, treatment (null) c = rng.normal(0, np.sqrt(n), (S, D)) # daily SUMS, control (null) ct, cc = np.cumsum(t, 1), np.cumsum(c, 1) N = np.arange(1, D+1) * n z = (ct/N - cc/N) / np.sqrt(2/N) # z path per sim peek = np.mean(np.any(np.abs(z) > 1.96, axis=1)) # stop at first crossing fixed = np.mean(np.abs(z[:, -1]) > 1.96) # only day-20 look print(round(peek, 5), round(fixed, 5)) # -> 0.25385 0.04915
python — library equivalent (Bonferroni floor) from statsmodels.stats.multitest import multipletests # Bonferroni is the conservative sequential fix; production uses gsDesign / Eppo / Statsig SDKs per_look_alpha = 0.05 / 20 # 0.0025 -> |z| > 3.02, valid but wasteful
Now the three fixes, in order of sophistication.
Fix 1 — alpha spending. Treat α = 0.05 as a budget spent across looks. Each interim look spends some alpha with a stricter threshold so the total false-positive probability stays 0.05. Bonferroni (0.05/20 per look) is valid but wasteful because looks are correlated; spending functions do it exactly.
Fix 2 — group-sequential designs. Pre-plan K looks with computed boundaries. O'Brien-Fleming style: nearly impossible to stop at look 1 (|z| > ~4.5 early), relaxing to ~1.96–2.04 at the final look — you keep almost all your final power but gain a legitimate early exit for huge effects or harms. Pocock style: the same threshold (~2.4 for 5 looks) at every look — easier early stops, costlier final threshold. Clinical trials have handled peeking this way since the 1970s.
Fix 3 — always-valid p-values / confidence sequences. Constructed so the guarantee holds at every moment simultaneously: "at no point in time will this sequence falsely reject more than 5% of the time." You may peek continuously and stop whenever you like. The price: wider intervals than a fixed-horizon test at any given n (the martingale construction pays for infinite looks). Netflix's 2024 engineering series describes exactly this for streaming-quality regressions; Eppo and Statsig ship it by default.
One asymmetry worth stating: peeking to STOP FOR HARM (guardrail metrics crashing) is good practice and is exactly what early-stopping boundaries formalize. It's peeking-to-declare-victory that manufactures winners.
Frame the constraint: stakeholders will monitor continuously, so choose methods whose guarantees survive it. Options: (a) group-sequential boundaries (O'Brien-Fleming spending) if experiments have planned horizons — near-full final power, principled early stops for huge wins or harms; (b) always-valid p-values / confidence sequences for open-ended monitoring — peek and stop anytime, at the cost of wider intervals; (c) at minimum, dashboard hygiene: show the corrected boundary, not raw p, and label interim reads as non-decisional. Trade-offs to name: power loss vs peeking freedom, cultural adoption (teams hate "the bar moved"), and asymmetric rules — loose for stopping on guardrail harm, strict for declaring victory. Cite that Netflix (2024) and commercial platforms (Eppo, Statsig) ship exactly this.
Peeking wastes your data by corrupting the stopping rule. But there's a way to get more signal out of the same users — legitimately, with no cost to validity — by subtracting noise you could have predicted before the experiment even started. That free lunch is CUPED, Chapter 5.
Chapter 3 said detecting small effects needs brutal sample sizes. But here is the secret: most of your metric's variance isn't noise from the treatment. It's users being persistently different from each other — and that part was knowable before the experiment started. Subtract it.
Diagnose where experiment noise comes from. In a sessions-per-user metric, the spread is dominated by stable user-to-user differences — power users vs occasional users — not by anything the treatment did. Randomization scatters these users across both arms, and their variance inflates the SE of the difference.
The key realization: anything measured BEFORE randomization is independent of treatment assignment. So adjusting the outcome by a pre-period covariate cannot bias the treatment effect — it can only soak up variance. This is the one free lunch in experimentation.
Enter CUPED (Controlled experiments Using Pre-Experiment Data; Deng, Xu, Kohavi & Walker, Microsoft 2013). Define the adjusted metric:
where X is the same metric measured in the pre-period and θ is exactly the OLS slope of Y on X. Subtracting θ(X − X̄) removes the predictable component; the group means are preserved because the correction has mean zero. And the payoff formula:
Read the meaning. A pre-period correlation of 0.7 removes 49% of variance (2× fewer users); 0.9 removes 81% (5× fewer); our worked example's 0.96 removes 92% (12.6× fewer). Retention-style metrics routinely have ρ of 0.5–0.8 week over week, so CUPED is a standing 2–4× traffic multiplier — which is why every major platform (Microsoft, Netflix, Airbnb, Statsig, Eppo) has it on by default.
Left: users scattered by pre-period metric X (horizontal) vs experiment metric Y (vertical), with the OLS line of slope θ drawn through. Right: two distributions of the effect estimate — raw Y-difference vs CUPED-adjusted. Raise the correlation slider and watch the cloud tighten around the line while the CUPED distribution visibly squeezes and the raw one stays fat. The readout shows θ, ρ², variance reduction, and equivalent extra users.
Now the full 5-user worked example, every arithmetic step. X = sessions in the 2 weeks BEFORE the experiment = [2, 4, 6, 8, 10]; Y = sessions during the experiment = [3, 5, 5, 9, 10]. Population variances (divide by n).
1. Means. X̄ = (2+4+6+8+10)/5 = 30/5 = 6. Ȳ = (3+5+5+9+10)/5 = 32/5 = 6.4.
2. Deviations. X−6 = [−4, −2, 0, 2, 4]; Y−6.4 = [−3.4, −1.4, −1.4, 2.6, 3.6].
3. Covariance. ((−4)(−3.4) + (−2)(−1.4) + 0(−1.4) + 2(2.6) + 4(3.6))/5 = (13.6 + 2.8 + 0 + 5.2 + 14.4)/5 = 36/5 = 7.2.
4. Variances. var(X) = (16+4+0+4+16)/5 = 40/5 = 8. var(Y) = (11.56+1.96+1.96+6.76+12.96)/5 = 35.2/5 = 7.04.
5. Theta. θ = cov/var(X) = 7.2/8 = 0.9. (Each pre-period session predicts 0.9 experiment sessions.)
6. Adjusted values Yi − 0.9(Xi−6): [3+3.6, 5+1.8, 5−0, 9−1.8, 10−3.6] = [6.6, 6.8, 5.0, 7.2, 6.4].
7. Mean preserved. (6.6+6.8+5.0+7.2+6.4)/5 = 32/5 = 6.4 — identical to Ȳ, so effect estimates stay unbiased.
8. New variance. deviations [0.2, 0.4, −1.4, 0.8, 0.0]; squares [0.04, 0.16, 1.96, 0.64, 0]; var = 2.8/5 = 0.56.
9. Theory check. ρ = 7.2/√(8×7.04) = 7.2/7.5047 = 0.9594; ρ² = 0.9205. Predicted var = 7.04×(1−0.9205) = 7.04×0.0795 = 0.56. Exact match.
10. Payoff. variance reduced 92.05%; required n scales with var, so n drops by 7.04/0.56 = 12.57× — a 300-day experiment becomes 24 days, zero extra users.
python — from scratch import numpy as np X = np.array([2,4,6,8,10.]) # pre-period metric (BEFORE randomization) Y = np.array([3,5,5,9,10.]) # in-experiment metric cov = np.mean((X - X.mean()) * (Y - Y.mean())) theta = cov / X.var() # 0.9 Yc = Y - theta * (X - X.mean()) # adjusted rho = cov / np.sqrt(X.var() * Y.var()) print(theta, Yc) # -> 0.9 [6.6 6.8 5.0 7.2 6.4] print(Yc.mean(), round(Yc.var(),4)) # -> 6.4 0.56 (mean preserved, var collapsed) print(round(rho**2,4), round(Y.var()/Yc.var(),2)) # -> 0.9205 12.57x fewer users
python — library equivalent import numpy as np theta, _ = np.polyfit(X, Y, 1) # slope == theta == 0.9 # equivalently: statsmodels smf.ols("Y ~ treat + X") — regression adjustment IS CUPED
Practicalities. New users have no pre-period — set their X to X̄ (no adjustment; CUPED gracefully degrades). The covariate needn't be the same metric, just correlated and PRE-treatment. θ is computed pooled across arms. And warn hard: adjusting by anything measured after randomization (a post-treatment variable) can absorb the treatment effect itself and bias the estimate — the pre-period boundary is sacred.
Place the siblings. Stratification randomizes within homogeneous buckets (platform, country, activity tier) so arms are balanced by construction — the same variance-reduction idea applied at assignment time rather than analysis time; post-stratification is its analysis-time twin. Paired / within-subject designs are the extreme: each unit is its own control (interleaving in Chapter 6 is this idea for rankers), removing between-user variance entirely at the cost of carryover risks.
Page-views during the experiment are post-treatment: if the treatment increases engagement, the covariate carries part of the causal effect, and regressing it out subtracts real signal — bias severe enough to flip signs. This is "conditioning on a mediator." Correct CUPED uses only pre-randomization data: same metric in a 1–4 week pre-period, θ = cov/var computed pooled, new users assigned X = X̄ (no adjustment). Sanity checks a staff engineer names: adjusted and raw means must agree in expectation (large discrepancy = leakage), A/A tests should show variance shrink but no effect, and the covariate window must end strictly before first exposure. Bonus: mention regression-adjustment equivalence and that multiple pre-period covariates generalize CUPED to a regression.
CUPED, stratification, and pairing all assume the core A/B machinery is valid — that one user's treatment doesn't touch another's outcome. In marketplaces, social networks, and shared-model systems, that assumption shatters, and the naive estimate can be off by 6× or point the wrong way. Chapter 6 fixes it.
Chapter 1's fine print — "no interference between users" — was doing a lot of quiet work. In a ride-sharing app, giving half the riders a better matching algorithm doesn't create two parallel worlds. It creates ONE world where treatment riders steal cars from control riders.
Name the assumption formally: SUTVA (Stable Unit Treatment Value Assumption) — each user's outcome depends only on THEIR assignment. Marketplaces, social networks, ranking systems with shared inventory, and ML models trained on pooled logs all violate it. The A/B test still runs; it just answers a corrupted question.
Work the marketplace cannibalization numbers. Ride-share matching, shared driver pool, baseline match rate 70%. The truth (god view): launching to 100% lifts the global rate to 72% — a +2pp effect. But in a 50/50 rider-split test, treatment riders match at 78% — partly by grabbing drivers that would have served control riders, who drop to 66%. Naive estimate = 78 − 66 = +12pp: six times the true +2pp launch effect.
A grid city with riders and a shared pool of cars. In User-split A/B, treatment riders get matching priority: their rate rises to ~78% while control's falls to ~66% below the 70% baseline — bars show the naive +12pp vs the dashed godview truth (+2pp). In Switchback, the whole city flips between arms on a randomized schedule; the timeline fills and the estimate converges near +2pp. Crank scarcity to zero and the bias vanishes — teaching WHEN user-split is fine.
The control arm was damaged by the treatment arm — the comparison measures "treatment vs a cannibalized control," not "launch vs no launch." Here is the full decomposition.
1. Naive user-split estimate = 78% − 66% = +12pp.
2. True launch effect = 72% − 70% = +2pp.
3. Overstatement factor = 12 / 2 = 6×. A finance model projecting +12pp onto full launch overstates incremental matches sixfold.
4. Decompose. Treatment uplift vs no-launch baseline = 78 − 70 = +8pp (real improvement plus stolen supply); control damage = 66 − 70 = −4pp (pure cannibalization). The naive diff ADDS the theft twice: (78−66) = (78−70) + (70−66) = 8 + 4 = 12.
5. Switchback readout. In treatment windows the whole city runs the new matcher → ~72%; in control windows ~70%. Estimate = 72 − 70 = +2pp — the launch-relevant number — with power now governed by the NUMBER OF WINDOWS (e.g. 336 half-hour windows/week), not riders.
6. Sign check. Had the feature had positive spillovers (referral rewards), control would be lifted above 70 and the naive diff would understate — interference bias has no universal direction.
Generalize the failure directions. Competition effects (ads, marketplaces) inflate estimates; positive spillovers (social features, referral loops) deflate them — your control users got some treatment benefit through the network, shrinking the gap. Either way the naive diff is wrong, and you can't even sign the error without thinking about the mechanism.
Design fix 1 — switchback experiments. When the interference is global but fast-mixing, randomize TIME WINDOWS instead of users: the whole city is treatment 7:00–7:30, control 7:30–8:00, flipping on a randomized schedule. Both arms see the whole marketplace; the unit of analysis is the window. Costs: temporal carryover (surge pricing at 7:29 leaks into 7:31 — leave washout gaps), far fewer effective units (windows, not users, so power drops), and within-window correlation. This is standard at DoorDash, Uber, Lyft; formalized in Bojinov, Simchi-Levi & Zhao (Management Science 2023).
Design fix 2 — cluster / graph-cluster randomization. For social interference, randomize connected communities together (whole friend-groups, whole geos) so most edges stay within-arm. Trade-off: fewer, lumpier units → higher variance; imperfect clustering leaves residual bias. LinkedIn/Meta run ego-cluster designs for feed and networking features.
Design fix 3 — interleaving for rankers. Instead of showing half the users ranker A and half ranker B, MERGE both rankings into one result list per query (team-draft interleaving: alternate picks like schoolyard captains) and see whose documents get clicked. Every query is its own paired comparison — within-subject in Chapter 5's language — making it 10–100× more sensitive than user-split A/B for ranking preference. Limits: it answers "which ranker do users prefer clicking," not "what happens to sessions/revenue" — search teams (Bing, Airbnb, Amazon) use interleaving to triage candidates, then A/B the winners.
python — from scratch (the decomposition) naive = 0.78 - 0.66 # user-split A/B true = 0.72 - 0.70 # godview launch effect print(round(naive,2), round(true,2), round(naive/true,1)) # -> 0.12 0.02 6.0 # theft counted twice: (78-70) + (70-66) = 8 + 4 print(round((0.78 - 0.70) + (0.70 - 0.66), 2)) # -> 0.12
python — library note # No one-liner exists — that IS the lesson. Analysis of a switchback: # window-level means with cluster-robust / HC standard errors: # import statsmodels.formula.api as smf # smf.ols("rate ~ arm", data=windows).fit(cov_type="cluster", cov_kwds=...) # Tooling: DoorDash/Uber internal switchback platforms.
Two time-based traps masquerade as effects. Novelty effect: users click the shiny new thing for a week, then regress. Primacy / change-aversion: users punish any change initially, then adapt. Both make week-1 readings systematically wrong in opposite directions. Detection: plot the daily treatment effect over time; a decaying or rising curve means the long-run effect isn't measured yet. Mitigation: run 2+ weeks, analyze only post-stabilization data, or keep a long-term holdout (5% never exposed for a quarter).
Map interference first: pricing shifts demand against a shared courier supply and can shift courier behavior city-wide → user-split is biased, so randomize time windows (switchback) within each city, or city-level cluster randomization if cross-window carryover is too strong. Window length trades carryover bias (short) against unit count/power (long); add washout buffers and randomize the schedule, stratified by hour-of-day and day-of-week. Analysis at the window level with cluster-robust SEs; power on the number of windows (Chapter 3 with n = windows, so variance reduction — Chapter 5 stratification by hour, CUPED on same-window-last-week — matters enormously). Guardrails: courier earnings, delivery times, cancellations. Staff extras: pre-registered novelty burn-in, a few holdout cities as long-term control, and an explicit statement of the estimand (global launch effect, not user-level effect).
Every design so far runs live on real users. But teams love to evaluate models offline first, on logged data, to avoid spending traffic. That backtest is seductive — and it routinely lies, because the logs were written by the old policy. Chapter 7 explains why, and how to partially fix it.
Your new recommender beats the old one by 8% on held-out clicks. You ship it. Online: flat. Nobody cheated. The offline eval answered a different question than the launch asked.
Diagnose the gap structurally, not as "noise." Offline evals score the new model on logs generated BY THE OLD MODEL. Four mechanisms follow. (1) Counterfactual blindness — you never observed what users would do with items the old policy never showed; the new ranker's favorite item may have zero logged impressions. (2) Feedback loops — logged clicks reflect old-policy position bias (top slots get clicked because they're top). (3) Metric mismatch — offline proxies (NDCG on clicks) vs online goals (sessions, retention). (4) System effects — latency, caching, UI interplay exist only online. (Which online metric to use is metrics-ladder's territory; here we own the statistical bridge.)
Make it vivid. A movie recommender trained and evaluated on logs where the old policy always pushed blockbusters. A new policy favoring niche films looks TERRIBLE offline — the logs contain almost no evidence about niche films — while possibly being better online. The logs are not the world; they're the old policy's shadow of it.
Enter off-policy evaluation (OPE): estimate the ONLINE value of a NEW policy from OLD logs, correcting for the mismatch. The key requirement: the logging policy must have been stochastic, and you must know its propensities (the probability it had of taking each logged action). Deterministic old systems make OPE impossible — which is a design argument for adding exploration/randomization to production systems NOW, so future models can be evaluated cheaply.
Inverse propensity scoring (IPS) in plain words: reweight each logged event by (probability NEW policy takes that action) / (probability OLD policy took it). Events the new policy loves but the old policy rarely tried get huge weights (rare evidence standing in for many counterfactual events); events the new policy would never take get weight zero. The weighted average of rewards is an unbiased estimate of the new policy's online value.
Logged event cards stream in (action A/B, its logging propensity, reward). Each passes through a reweighter where its importance weight w = πnew/πlog scales the card — weight-5 cards balloon, weight-0 cards shrink to dust. A running IPS needle wobbles as heavy cards land. Watch the effective sample size meter reveal how few events actually carry the estimate. Shrink the logging policy's exploration and the weights explode; raise the weight cap to tame them (at the cost of bias).
Now the 5-event worked example, end to end. Logging policy showed item A with probability 0.8 and item B with 0.2 (stochastic logger, propensities recorded). Five logged events (action, propensity, reward): (A, 0.8, 1), (A, 0.8, 0), (B, 0.2, 1), (A, 0.8, 1), (B, 0.2, 0). Evaluate the target policy "always show B."
1. Logging policy's own value (naive log average) = (1+0+1+1+0)/5 = 3/5 = 0.6.
2. Importance weights wi = πnew(ai)/πlog(ai). Target always-B puts probability 1 on B, 0 on A. Events 1,2,4 (action A): w = 0/0.8 = 0. Events 3,5 (action B): w = 1/0.2 = 5.
3. IPS estimate = mean(wi ri) = (0·1 + 0·0 + 5·1 + 0·1 + 5·0)/5 = 5/5 = 1.0.
4. Read it critically. The estimate rests entirely on TWO logged B-events, one rewarded, one not. Each speaks for 5 events. Effective sample size = (Σw)²/Σw² = 10²/50 = 2 — we effectively have two data points.
5. Weight capping at c=3. capped weights [0,0,3,0,3]; estimate = (3·1 + 3·0)/5 = 3/5 = 0.6. The explosion is tamed but the estimate is now biased toward the logging policy's value (here it lands exactly on 0.6 — the cap threw away precisely the counterfactual correction).
6. Moral. unbiased IPS said 1.0 with ESS=2; capped said 0.6. The true value of always-B is unknowable from 5 events — the honest conclusion is "collect more exploration on B or run the online test."
Name the disease and the treatments. Variance explodes as the policies diverge (weights = ratio of policies). Weight capping trades unbiasedness for sanity. Self-normalized IPS divides by the sum of weights. Doubly robust estimators combine a reward MODEL with IPS on the model's residuals: unbiased if EITHER the model or the propensities are right, and lower variance. Rule of thumb: ESS = (Σw)²/Σw²; if it's a tiny fraction of n, your OPE is decorative.
python — from scratch import numpy as np log = [('A',0.8,1), ('A',0.8,0), ('B',0.2,1), ('A',0.8,1), ('B',0.2,0)] # target policy: always B -> pi_new(B)=1, pi_new(A)=0 w = np.array([(1.0 if a=='B' else 0.0) / p for a,p,r in log]) r = np.array([e[2] for e in log]) naive = r.mean() # 0.6 (logging policy's value) ips = (w * r).mean() # 1.0 (unbiased, high variance) capped = (np.minimum(w, 3) * r).mean() # 0.6 (biased toward logger) ess = w.sum()**2 / np.sum(w**2) # 2.0 (only 2 effective events!) print(naive, w.tolist(), ips, capped, ess) # -> 0.6 [0,0,5,0,5] 1.0 0.6 2.0
python — library equivalent # Open Bandit Pipeline productizes IPS/SNIPS/DR: # from obp.ope import OffPolicyEvaluation, InverseProbabilityWeighting import numpy as np ips = np.mean(w * r) # the estimator itself is a one-liner
So is offline eval useless? No — answer the correlation question. The mature practice is to CALIBRATE the offline-online relationship empirically: for every model you've A/B tested, plot offline gain vs online gain. If the scatter shows rank-correlation, offline evals are a valid FILTER (kill bad candidates cheaply) even when the magnitudes don't transfer. If it's a cloud, your offline metric is measuring the wrong thing (send that finding to metric-design). Industry reality: offline-online correlation is often weak for recommenders/search and must be re-established per surface.
The workflow top teams converge on is a pyramid: offline eval (cheap, biased) → OPE on logged exploration data (medium cost, unbiased-but-noisy) → interleaving (Chapter 6, sensitive preference test) → full A/B (expensive ground truth) → long-term holdout. Each rung answers a more launch-shaped question. The GPT-4o sycophancy incident (OpenAI postmortem, May 2025) is the cautionary tale: offline evals AND short-run A/Bs looked fine while expert judgment flagged the failure — no rung of the pyramid substitutes for the others.
Diagnose in layers: (1) plot historical offline-gain vs online-gain per launched model — if rank-correlation has decayed, the offline metric is saturated or misaligned (position bias in click labels, distribution shift since the eval set was frozen); (2) check policy overlap — if candidate models now diverge more from production, logged-data evals lose validity (compute ESS of importance weights); (3) check metric mismatch — NDCG-on-clicks vs online session success. Fixes by leverage: inject logged randomization into production (small epsilon or Plackett-Luce sampling) to enable honest OPE with recorded propensities; adopt doubly robust estimators over raw IPS; insert interleaving between offline eval and A/B as a cheap sensitive filter; and periodically re-fit the offline-online calibration curve, treating offline eval explicitly as a candidate FILTER, not a launch predictor. The failure is common enough that mature orgs re-validate the correlation quarterly.
You now know the traps one at a time — confounding, selection, SRM, power, peeking, interference, offline drift. The payoff chapter assembles them into one god console: set the truth yourself, and watch honest and dishonest pipelines report on the world you made. Chapter 8.
You now know the traps individually. This chapter hands you a god console: YOU set the true effect, and then you get to watch honest and dishonest experiment pipelines report on the world you created.
Here is the simulator's philosophy. In real life you never know the truth, which is why bad practice survives — a 25%-false-positive pipeline FEELS productive because it ships "winners." In the simulator the truth is a slider, so every methodological sin becomes immediately visible as a wrong-verdict count.
A three-panel god console. Top: world controls + a live power readout. Middle: the current experiment's z-walk with its active stopping boundary, verdict stamp, and CI whiskers. Bottom scoreboard: across all runs — false-positive rate (truth=0), power (truth>0), average estimated lift among WINs vs the dashed true lift (the winner's-curse gap), SRM catches, and a "shipped features that do nothing" tally. Set truth = 0 with peeking on and watch a quarter of experiments "win"; flip to a sequential boundary and watch it snap back to 5%.
A guided tour. Station 1 — the world: set baseline conversion (default 5%), true lift (0 to +2pp, including exactly 0), daily traffic (default 20k), horizon (default 14 days). The god panel shows Chapter 3's power calculation live: "with these settings, an honest fixed-horizon test has X% power." Station 2 — the policy switchboard: fixed-horizon vs daily-peeking (Chapter 4) as the master toggle; a CUPED toggle (Chapter 5, ρ = 0.7) that visibly tightens CIs and raises power; an "Inject SRM" button (Chapter 2) that silently drops 2% of one arm and stamps a chi-square FAIL. Station 3 — the run view: each experiment renders as Chapter 4's z-walk with its boundary; verdicts stamp WIN / NO EFFECT / SRM-VOID; batch mode fills the scoreboard.
Do these scripted discoveries in order. (1) truth = 0, fixed horizon → ~5% wins, as promised; flip to daily peeking → ~25% wins — nothing about the world changed. (2) truth = +0.3pp (below the configured MDE), fixed horizon → power ~15–20%; most runs say NO EFFECT and you feel "underpowered." Toggle CUPED and watch power jump. (3) Inject SRM with truth = 0 and see biased "lifts" appear alongside the FAIL banner — the chi-square catches what the effect estimate can't reveal about itself. (4, the payoff) with truth = +0.5pp and peeking on, look at the average ESTIMATED lift among winners — noticeably above 0.5pp. Winners are selected noise-peaks; even "real" wins are exaggerated (Chapter 10).
Two sanity numbers the scoreboard must reproduce, computable by hand.
1. Expected false winners among 20 null experiments (honest α = 0.05) = 20 × 0.05 = 1.0 — even a fully honest quarterly portfolio ships about one dud.
2. P(at least one false winner) = 1 − (1 − 0.05)20 = 1 − 0.9520 = 1 − 0.3585 = 0.6415. Nearly two-thirds of quarters contain a false win somewhere — portfolio thinking, not per-test thinking.
3. Power cross-check. With n = 8,155/arm, p1 = 5%, p2 = 6%, detection probability is 80% by construction — the simulator's empirical win-rate over truth=+1pp batches must converge near 0.80. Monte Carlo tolerance at 200 runs: SE = √(0.8×0.2/200) = 0.028, so 77–83% is in-spec.
4. Peeking check. the same 200-run batch with daily peeking on truth=0 must land near 25.4% (Chapter 4); SE = √(0.25×0.75/200) = 0.031.
python — from scratch (the simulator core) import numpy as np def run_experiment(true_lift, base, n_day, days, policy, rng): zc = 1.96 ct = cc = 0.0; verdict = "NO EFFECT"; est = 0.0 p_c, p_t = base, base + true_lift for d in range(1, days+1): ct += rng.binomial(n_day, p_t); cc += rng.binomial(n_day, p_c) N = d * n_day pt, pc = ct/N, cc/N; est = pt - pc pool = (ct+cc) / (2*N) se = np.sqrt(pool*(1-pool)*2/N) + 1e-12 z = est / se bound = zc if policy=="peek" else (zc if d==days else 1e9) if policy=="seq": bound = zc * np.sqrt(days/d) # O'Brien-Fleming-ish funnel if abs(z) > bound: verdict = "WIN" if est > 0 else "NO EFFECT"; break return verdict, est def batch(true_lift, policy, n=200): rng = np.random.default_rng(0) wins = [run_experiment(true_lift/100, 0.05, 10000, 14, policy, rng) for _ in range(n)] w = [e for v,e in wins if v=="WIN"] print(policy, "win-rate", len(w)/n, "mean est|win", np.mean(w) if w else 0) batch(0.0, "fixed") # ~0.05 FPR batch(0.0, "peek") # ~0.25 FPR batch(1.0, "fixed") # ~0.80 power; mean est|win slightly > 1.0 (winner's curse)
Three compounding mechanisms, all visible in this chapter's simulator: (1) false positives — some fraction of the 80 are truth-zero experiments that crossed alpha (with peeking, possibly many); (2) winner's curse — conditioning on significance biases estimated lifts upward even for real effects, so summing winners' estimates systematically overshoots; (3) interaction/decay — effects measured in isolation don't stack additively and novelty fades (Chapter 6). The holdout is the ground truth for the PORTFOLIO. Fixes: sequential-testing discipline, shrinkage/empirical-Bayes adjustment of winner estimates before roadmap math, long-term holdouts as standing infrastructure, and reporting "holdout-validated" gains to leadership. This separates candidates who know tests from candidates who run programs.
The simulator taught the machinery. But experimentation interviews test something the console can't: whether you reason about design validity before statistics. Chapter 9 drills the full staff-level question bank.
Every question in this chapter has been asked, in some form, in real staff+ interviews at experimentation-heavy companies. The pattern behind all of them: the interviewer names a tempting shortcut and watches whether you can price its cost.
How these interviews are structured: a case ("we want to test X"), then escalating complications — small traffic, a marketplace, an impatient stakeholder, a suspicious result. The rubric is usually validity-first: candidates who jump to test statistics before checking assignment, unit, interference, and SRM fail early.
Drill the triage order as a checklist you can say out loud: (1) What's the unit and does it match the metric? (2) How is assignment done, and did SRM pass? (3) Could arms interfere? (4) Was the horizon and stopping rule pre-registered? (5) Only then: what does the CI say? Practicing this order converts panic into procedure.
A flashcard scenario machine deals a randomized experiment card (unit / assignment / metric / complication). Tap the FIRST validity check that fails, from the 6-button rubric. Correct taps build a streak; wrong taps flip the card to reveal the reasoning and the chapter to revisit. The rubric-order strip up top reinforces the sequence.
The single most-quoted staff computation is the portfolio false-discovery rate. 100 experiments, 10% with true effects, 80% power, α = 5%. Being able to run this on a whiteboard, then discuss how peeking makes it worse, is a hiring signal by itself.
1. Split. True-effect experiments = 100 × 0.10 = 10. Null experiments = 90.
2. True winners = 10 × power = 10 × 0.80 = 8.
3. False winners = 90 × α = 90 × 0.05 = 4.5.
4. Total winners = 8 + 4.5 = 12.5.
5. FDR among shipped winners = 4.5 / 12.5 = 0.36 — thirty-six percent of your "wins" are duds, at PERFECT hygiene.
6. Stress it. With peeking inflating α to 0.25 (Chapter 4): false winners = 90 × 0.25 = 22.5, FDR = 22.5/(8+22.5) = 0.738 — three of four shipped winners do nothing. And at a 5% base rate: FDR = 4.75/(4+4.75) = 0.543.
Discussion this unlocks: base rates of good ideas dominate; raising idea quality and power beats tightening alpha; holdouts measure the realized portfolio.
python — from scratch def fdr(n, base, power, alpha): true_win = n * base * power false_win = n * (1 - base) * alpha return false_win / (true_win + false_win) print(round(fdr(100, 0.10, 0.8, 0.05), 3)) # 0.36 honest hygiene print(round(fdr(100, 0.10, 0.8, 0.25), 3)) # 0.738 with peeking print(round(fdr(100, 0.05, 0.8, 0.05), 3)) # 0.543 at 5% base rate
python — library equivalent from statsmodels.stats.multitest import multipletests # Benjamini-Hochberg controls FDR across a batch of experiment p-values: # reject, p_adj, _, _ = multipletests(pvals, alpha=0.05, method="fdr_bh")
Rapid-fire numeric fluency to rehearse: 4× sample for half the MDE; ρ² variance reduction from CUPED; 1 − 0.95k for portfolio false-win probability; ESS = (Σw)²/Σw² for OPE trustworthiness; χ² = 2·(δ²/E) for a two-arm SRM check.
Each question maps back to the chapter that teaches its machinery, so a wrong answer is a link to re-study, not just a red X. Expand each for the model answer and a follow-up.
Unit: user (consistent experience; ranking changes are visible), via hash(user_id + experiment_salt) into buckets — deterministic, sticky, decorrelated from other experiments. Primary metric pre-registered at the user level (e.g. purchase rate per user over the window) so the analysis unit matches the randomization unit; session-level metrics would need delta-method corrections. Power: pick the MDE as a business decision, compute n/arm with the two-proportion formula, then duration = total n / eligible daily traffic, rounded up to full weeks to absorb day-of-week cycles. Pre-register horizon and stopping rule; run an SRM chi-square before reading metrics; segment checks (platform, new/returning) for differential loss. If traffic is tight, CUPED with a pre-period purchase covariate cuts required sample by roughly 1 − ρ². I'd also consider interleaving as a cheaper preference filter before spending A/B traffic, since ranking changes are exactly where interleaving is 10–100× more sensitive.
Follow-up: Your data scientist proposes randomizing by query instead of by user to get more units. What breaks? (Answer: the same user sees both rankers across queries — inconsistency is its own treatment — and per-user metrics no longer match the unit; query-level SEs also ignore within-user correlation.)
First establish whether this is a scheduled interim look or opportunistic peeking. If the team has watched daily and this is the first dip below 0.05, the effective false-positive rate across those looks is several times 5% (Chapter 4), and the +estimate is selection-inflated. My default: complete the planned horizon and evaluate once, as pre-registered. If early decisions are a recurring business need, fix the platform rather than the norm: group-sequential boundaries or always-valid p-values give legitimate early stopping with controlled error — under an O'Brien-Fleming boundary, day-11 evidence would need to be far stronger than z = 2.05. I'd also flag the asymmetry: stopping early for guardrail HARM is always allowed; stopping early to declare victory is the move that manufactures false winners.
Follow-up: The PM says "the p-value is what it is — how can looking at it change anything?" Two-sentence rebuttal: Looking changes nothing; stopping does. The rule "quit at the first p < 0.05" selects the noise's maximum excursion, so the reported p is drawn from a 25%-false-positive process, not a 5% one.
SRM is a statistically significant deviation of observed assignment counts from the designed ratio, tested with a chi-square on the counts — e.g. 50,600 vs 49,400 on a planned 50/50 of 100k gives χ² = 14.4, p = 0.00015. It's dangerous because the imbalance almost never comes from the random coin; it comes from units being LOST differentially after assignment — crashed clients, bot filters, redirect drops — and lost users are never a random subset. Every metric is then computed on selectively surviving populations, so the effect estimate is biased in an unknown direction and magnitude. Reweighting can't fix it because the problem isn't group SIZE, it's group COMPOSITION: you'd be reweighting the survivors, not resurrecting the missing. Correct responses: diagnose (segment the SRM by platform, browser, country to localize the loss), fix the mechanism, rerun. Microsoft's ExP reported ~6% of experiments trip SRM, so this is an everyday gate.
Follow-up: Where in the pipeline can SRM be introduced even before assignment, and how would a per-segment SRM check reveal it? (Answer: an upstream filter or eligibility check that fires differently by arm/segment; segmenting the chi-square localizes the loss to, say, one browser or country.)
PM version: most of the noise in our metric is that users differ from each other in stable ways we could already see before the experiment — power users vs casual users. CUPED subtracts that predictable part, so the same experiment gets the precision of one several times larger; a typical retention metric needs half or a third of the traffic, with no change to what we're measuring. Statistician version: define Yadj = Y − θ(X − X̄) with X a pre-treatment covariate and θ = cov(X,Y)/var(X), the OLS slope; since X is independent of assignment, the adjustment is mean-zero in both arms and the ATE estimate stays unbiased, while var(Yadj) = var(Y)(1 − ρ²). It is numerically regression adjustment with a pre-period covariate; multiple covariates generalize it; new users take X = X̄ and degrade gracefully. The one hard rule: the covariate window must end strictly before first exposure, or you're conditioning on a mediator and can bias the estimate, potentially flipping signs.
Follow-up: Why does CUPED not help at all for a brand-new user, and what would you use instead? (Answer: no pre-period exists, so X = X̄ means zero adjustment; use stratification on signup cohort or an early in-session pre-exposure signal that still precedes treatment.)
Choose switchbacks when treatment and control units share a finite resource or fast-mixing global state — marketplace supply, surge pricing, fleet positioning — so user-split arms interfere and the naive difference measures "treatment vs cannibalized control" rather than the launch effect. In our ride-share example that inflation was sixfold: +12pp vs a true +2pp. Switchbacks randomize time windows for the whole system, so both arms experience the entire marketplace. You give up: statistical power (units are windows — hundreds, not millions; stratification by hour-of-day becomes essential), carryover robustness (state leaks across window boundaries; washout buffers or longer windows, formalized in Bojinov et al., Management Science 2023), and per-user measurement. Window length is the core knob: short = more units, more carryover; long = the reverse.
Follow-up: Your switchback shows a +2% effect but windows are strongly autocorrelated. How do you compute an honest standard error? (Answer: cluster-robust / HAC standard errors at the window level, or a block-bootstrap over contiguous window blocks; the naive i.i.d. SE understates uncertainty.)
Candidates: (1) counterfactual blindness — offline scoring used old-policy logs that barely explored the items the new model promotes, so the offline number was computed where the models overlap, not where they differ; (2) label bias — clicks encode the old ranker's position bias; (3) metric mismatch — NDCG-on-clicks vs online session success; (4) the online test may be underpowered for a small true effect (check the MDE math before declaring "no effect"); (5) system effects like latency eating the ranking gain. Pipeline: offline eval as a cheap FILTER whose offline-online correlation is empirically re-fitted from past launches; production exploration (logged stochasticity with recorded propensities) enabling honest OPE with doubly robust estimators and an ESS gate; interleaving as the sensitive preference test; A/B as ground truth for finalists; a long-term holdout for portfolio truth.
Follow-up: Production serving is deterministic today. What minimal change makes future off-policy evaluation possible, and what does it cost? (Answer: log a small amount of randomized exploration with recorded propensities — epsilon-greedy or Plackett-Luce sampling; cost is a slight short-term metric dip on the explored fraction.)
Three mechanisms stack: false positives (even honest α = 0.05 across many null experiments ships duds — peeking history multiplies that), winner's curse (conditioning on significance selects upward noise: at 50% power a true 1% lift reports 1.39 on average, so summed winner estimates systematically overshoot even when every effect is real), and non-additivity (effects interact, cannibalize, and decay as novelty fades). The holdout is the portfolio's ground truth and half is a typical outcome. Reporting fixes: shrink individual estimates via empirical Bayes toward the historical effect distribution before any roadmap math; publish "holdout-validated" impact quarterly alongside per-experiment claims; track the org's replication rate for large surprising wins; set power standards so the curse is structurally smaller. The cultural point matters most: a single experiment is an estimate, the holdout is the audit.
Follow-up: Sketch how you'd choose the holdout size — what's the power calculation for detecting that the stacked effect is half of claimed? (Answer: treat "stacked effect" as one aggregate metric; power to detect a 50% shortfall depends on its variance — size the holdout so the CI on aggregate impact excludes the summed-claim value.)
True winners: 10 × 0.8 = 8. False winners: 90 × 0.05 = 4.5. Winners total 12.5, so FDR = 4.5/12.5 = 36% — with perfect hygiene. Levers by leverage: the base rate of good ideas (better upstream research and offline filtering raises the 10% — at a 30% base rate FDR drops to ~13%), power (raising 0.8 toward 0.95 grows true wins), and only then alpha (tightening to 0.01 cuts false wins but taxes power everywhere). If the org peeks, alpha is effectively 0.2–0.3 and the FDR flips to majority-false — the single strongest argument for sequential-testing discipline. This is Ioannidis' "why most published findings are false" arithmetic transplanted to product experimentation.
Follow-up: Your org's replication rate for re-tested winners is 55%. Back out the implied base rate of real effects. (Answer: replication rate ≈ fraction of winners that are true = 1 − FDR; set 1 − FDR = 0.55 and solve the FDR equation for base rate given your power/alpha.)
Interference through the social graph: a control user with treated friends RECEIVES reactions — partial treatment — so the control arm no longer estimates the no-launch world and the naive difference typically understates the true effect (positive spillover is the mirror image of marketplace cannibalization, which overstates). SUTVA fails along every friendship edge crossing arms. Design: graph-cluster randomization — partition the social graph into communities that minimize cut edges, randomize whole clusters, analyze at the cluster level with cluster-robust errors; ego-cluster designs (randomize a focal user plus their neighborhood) when full partitioning is impractical. Costs: far fewer effective units, so power drops and stratification by cluster size/activity becomes essential, and imperfect clustering leaves residual bias proportional to cross-arm edges. I'd also estimate the spillover by comparing control users with many vs few treated friends — a validity check and a product insight about network effects.
Follow-up: How would you decide between geo-clusters and graph-clusters for this feature? (Answer: use the interference structure — if influence flows along explicit friendship edges, graph-clusters minimize cut edges; if it flows through shared physical/market context, geo-clusters are simpler and enough.)
Know both sides — interviewers often assign you the side you didn't volunteer.
FOR: peeking is behaviorally irresistible; hiding the number is the only intervention that survives incentives — teams ship at the first green digit regardless of training, and the 25%-FPR arithmetic is invariant to good intentions; guardrail metrics can stay visible for harm without exposing the primary. AGAINST: opacity destroys trust and teams rebuild shadow dashboards from raw logs; the correct fix is showing STATISTICALLY VALID interim results (sequential boundaries, always-valid CIs) so peeking becomes harmless by construction; hiding data also delays legitimate early stops for large effects and harms, which have real user cost.
FOR: logged data reflects the old policy's choices, so offline gains are systematically uninformative about the actions that make a new model different (the overlap/ESS argument); history at search and recsys companies shows offline-online correlation repeatedly decaying to noise; only randomized exposure measures value. AGAINST: this throws away the only scalable filter — you cannot A/B thousands of candidates, and a calibrated offline metric with periodically re-fitted correlation demonstrably rank-orders candidates; with logged exploration and doubly robust OPE, "offline" evidence approaches causal validity at a fraction of the cost; the fix is investing in the offline pipeline, not demoting it.
FOR: underpowered testing is worse than nothing — it mostly returns nulls that get misread as "no effect," and its rare significant results are winner's-curse-inflated by 40%+; small orgs get more decision value from user research, staged rollouts with guardrails, and diff-in-diff on natural variation. AGAINST: the dichotomy is false — variance reduction (CUPED, stratification, paired designs), sequential tests, and metric surrogates make small-traffic experiments honest at realistic MDEs; qualitative judgment has unbounded, undiagnosable bias, while an underpowered CI at least reports its own width; pre-registration and SRM discipline have cultural value beyond any single result.
Triage: (1) Why 11 of 14 — stopped early on significance? If monitored daily, the effective FPR is far above 5% and +2.1% is selection-inflated. (2) SRM chi-square, overall and by segment. (3) Unit-metric coherence: revenue per WHAT, randomized by WHAT; heavy-spender outliers argue for winsorization or user-level analysis. (4) Interference plausibility for the surface. (5) Was +2.1% the pre-registered primary or one of thirty dashboards (multiple comparisons — eval-statistics)? (6) Novelty: does the daily effect decay? Only after these: the CI, the power context, and often "complete the planned horizon, then decide." Interviewers score the ORDER and completeness.
One last trap survives even perfect practice, and it operates on your whole roadmap: condition on an experiment being significant, and its estimated effect is biased upward. Chapter 10 quantifies the winner's curse, maps the quasi-experimental toolbox for when you can't randomize, and places this lesson in the family.
One last trap, and it survives perfect practice: condition on an experiment being significant, and its estimated effect is biased upward — the same selection logic as peeking, operating on your whole roadmap.
Quantify the winner's curse. True lift 1.0%, standard error 0.5% (a 50%-power experiment — realistic for small effects). Among experiments that reach significance, the average estimated lift is 1.39% — a 39% exaggeration, from selection alone, with zero malpractice. The lower the power, the worse the curse: underpowered fields don't just miss effects, they systematically exaggerate the ones they find.
Experiments rain down as dots, each an observed lift drawn around the true effect (slider). A vertical significance gate splits them: non-significant fall grey below; significant stack into a green histogram to the right. Two markers track truth vs the running mean of the GREEN dots — the gap IS the curse, live. Drag the true effect down (power drops) and watch the green mean float far above truth. Toggle shrinkage to pull it back.
Now the closed-form worked example. True lift μ = 1.0%, SE σ = 0.5%, one-sided significance at z = 1.96 (threshold = 1.96 × 0.5 = 0.98%). Use the truncated-normal mean E[X | X > t] = μ + σ · φ(a)/(1 − Φ(a)) with a = (t − μ)/σ.
1. Threshold. t = 1.96 × 0.5 = 0.98% — "significant" when observed lift exceeds 0.98%.
2. Standardize. a = (t − μ)/σ = (0.98 − 1.0)/0.5 = −0.04.
3. Power. P(significant) = 1 − Φ(−0.04) = Φ(0.04) = 0.516 — a coin-flip, 52% power (realistically underpowered for a 1% effect).
4. Density & survival. φ(−0.04) = 0.3986; 1 − Φ(−0.04) = 0.5160.
5. Inverse Mills ratio. φ(a)/(1 − Φ(a)) = 0.3986/0.5160 = 0.7726.
6. Conditional mean. E[estimate | significant] = μ + σ × 0.7726 = 1.0 + 0.5 × 0.7726 = 1.386%.
7. Exaggeration. 1.386/1.0 = 1.39 — the average REPORTED win overstates the true effect by 39%, from pure selection. At 80% power (SE ≈ 0.357 for the same effect) the inflation is materially smaller — power is the antidote.
python — from scratch + closed form import numpy as np from scipy import stats mu, se = 1.0, 0.5 t = 1.96 * se # Monte Carlo rng = np.random.default_rng(0) est = rng.normal(mu, se, 100000) print(round(est[est > t].mean(), 3)) # -> ~1.386 # Closed form (truncated normal) a = (t - mu) / se E = mu + se * stats.norm.pdf(a) / stats.norm.sf(a) print(round(t,3), round(a,3), round(stats.norm.sf(a),4), round(E,4), round(E/mu,4)) # -> 0.98 -0.04 0.516 1.3863 1.3863
python — library equivalent from scipy import stats import numpy as np mu, se = 1.0, 0.5; t = 1.96 * se print(stats.truncnorm(a=(t-mu)/se, b=np.inf, loc=mu, scale=se).mean()) # -> 1.3863
Prescriptions: report shrunken estimates (empirical Bayes toward the org's historical effect distribution), replicate big surprising wins before roadmap math, power experiments properly (the curse shrinks as power rises — at 80%+ power for the TRUE effect the inflation is modest), and validate portfolios with long-term holdouts (Chapter 8).
Legal, ethical, or technical constraints sometimes forbid randomization. These methods move the burden from design to argument — each buys causality with an assumption you must defend.
| Method | Idea | Assumption you must defend |
|---|---|---|
| Difference-in-differences | Compare the treated group's before→after change to an untreated group's | Parallel trends: the two groups would have moved together absent treatment |
| Interrupted time series | Model the pre-trend, project it forward as the counterfactual | The trend model captures everything but the intervention (no coincident shocks) |
| Regression discontinuity | Units just above/below an arbitrary cutoff are as-if random | No manipulation of the running variable around the threshold |
| Instrumental variables / encouragement | Randomize the NUDGE when you can't randomize the treatment | The instrument affects the outcome ONLY through the treatment |
| Synthetic control | A weighted blend of unaffected units mimics the treated unit pre-launch | The pre-period fit implies a valid post-period counterfactual |
Link, don't duplicate. Each sibling owns its layer.
| Lesson | Why it connects |
|---|---|
| The Statistics of Evaluation | Prerequisite: every z-test, CI, and alpha here is built there — this lesson supplies the DESIGN those statistics assume. |
| Metric Design | Which metric deserves to be the experiment's primary/OEC is its territory; we take the metric as given and test it causally. |
| Plots That Tell the Truth | Presenting experiment readouts (CIs, daily-effect curves, segment forest plots) without lying is its machinery. |
| Regression Testing & Quality Gates (sibling — forthcoming) | Experiment verdicts feed release gates; it owns the CI/monitoring machinery downstream of a ship decision. |
| The Metrics Ladder (sibling — forthcoming) | Translating a +0.4pp lift into revenue narrative and exec reporting is the layer above this lesson. |
| Evaluating Generative Models (sibling — forthcoming) | When the outcome is generated text/images, the metric itself needs its pass@k / preference / judge-bias math before it can be A/B tested. |
| AI Evaluation (harness) | Covers the LLM eval PROCESS layer (judges, regression suites, A/B overview); this lesson owns the statistical engine underneath its A/B section. |
| Offline RL | Chapter 7's inverse propensity scoring is the doorway to full off-policy reinforcement learning from logged data. |
| Off-Policy Actor-Critic | Importance weighting between behavior and target policies — the same ratio as Chapter 7's IPS — powers off-policy RL. |
| Testing & Deployment | Canary and staged deployment infrastructure — operationally an experiment whose primary metric is "nothing broke" — lives there. |
| Symbol | Meaning | When it matters |
|---|---|---|
| δ (MDE) | smallest effect worth detecting | Sets sample size; write it in ABSOLUTE points |
| α / β | false-positive / false-negative rate | α = ship-a-dud risk; power = 1−β |
| ρ | pre-period correlation | CUPED removes ρ² of variance — note the SQUARE |
| w = πnew/πlog | importance weight | OPE variance explodes when policies diverge; watch ESS |
Lead with difference-in-differences: compare Germany's revenue change (Feb vs March) against a control set of similar non-launched countries; the identifying assumption is parallel trends — defend it with pre-period plots showing Germany and controls moving together for months. If no single good control exists, synthetic control: fit a weighted combination of countries that tracks Germany pre-launch, use its post-launch trajectory as the counterfactual, with placebo-in-space/time tests. Failure modes to volunteer: simultaneous German-specific shocks (the exact confounders Chapter 0 warned about), anticipation effects before March 1, spillovers if countries share supply/budgets, and the honest caveat that assumptions replace randomization — quantify sensitivity rather than presenting one number. Strong candidates propose engineering future escape hatches: staged regional rollouts with randomized ORDER give quasi-random variation next time.