The average salary at your company is $68k, and almost nobody earns near it. This lesson takes apart the one-number summaries we crush distributions into — what a percentile actually is, why the mean lies on skewed data, and which summaries you can trust when the data gets messy.
Ten support tickets came in last week. The time to resolve each one, in minutes, was: 2, 3, 3, 4, 4, 5, 5, 6, 7, and 120. Nine of them were quick. One was a two-hour outage. Your dashboard wants a single "average response time" for the SLA report, and everyone in the room reaches for the same number by reflex: the mean.
The mean — add every value, divide by how many there are — comes out to 15.9 minutes. That number is technically correct and simultaneously a lie about what your service actually feels like. This whole chapter is about breaking the reflex that says "average" is a safe, neutral summary of any pile of numbers.
Line up all ten values on a number line and mark where 15.9 falls. It lands in the empty gap between 7 and 120 — no ticket took anywhere near 15.9 minutes. A summary that sits where zero data points live isn't summarizing your data. It's summarizing the one outlier.
Now meet the alternative. The median is, informally, "the middle value when you line them all up." With ten tickets the two middle ones are 4 and 5, so the median is their average, 4.5 minutes — squarely inside the fast cluster where nine of your ten tickets actually live. Same data. Two summaries. Two completely different stories about your service.
Here is the reveal that motivates everything that follows. Grab that 120-minute ticket in the simulation below and drag it out further — to 500, then 1000. Watch the mean chase it upward: 15.9, then 53.9, then 103.9.
Now watch the median. It stays frozen at 4.5, no matter how far you drag. The mean is anchored to every value at once; the median only cares about the middle rank, so it never notices how extreme the extreme gets.
Name the property without a single formula. The median is robust: one wild value can't move it much. The mean is sensitive: one wild value can move it arbitrarily far. Neither is universally "better" — but reporting only the mean on skewed data, the way almost every dashboard does, systematically overstates the typical case.
This isn't a one-off trick with support tickets. Incomes are the textbook case: a handful of very high earners pull the "average income" well above what a typical person makes, which is exactly why economists and the census report median household income, not mean. Latency, file sizes, and machine-learning error distributions all share this lopsided, right-leaning shape — and on all of them, the mean quietly reports the tail instead of the typical case.
So here is the mission of the whole lesson. A distribution — a full set of numbers — is a rich object, and we routinely crush it down to one or two numbers to fit a report, a slide, or an SLA. This lesson teaches which numbers to crush it to, what each one throws away, and exactly when each one lies. Get this wrong and you ship the wrong SLA, hire against the wrong salary, or believe the wrong benchmark.
Ten ticket-time dots: nine clustered near 4 minutes, one draggable outlier. The MEAN needle glides to follow the outlier; the MEDIAN needle barely moves. Drag the far dot anywhere from 5 to 1000, or hit race to watch the mean chase and return on its own.
Ticket resolution times (minutes): [2, 3, 3, 4, 4, 5, 5, 6, 7, 120].
Let's build the mean and median from scratch so there's no magic — just sorting and arithmetic. Then we'll confirm the same numbers come out of numpy.
python — from scratch def mean(xs): return sum(xs) / len(xs) def median(xs): s = sorted(xs) # the single most-forgotten step n = len(s); mid = n // 2 if n % 2: # odd → the exact middle value return s[mid] return (s[mid - 1] + s[mid]) / 2 # even → average the two middles tickets = [2, 3, 3, 4, 4, 5, 5, 6, 7, 120] for out in (120, 500, 1000): d = tickets[:-1] + [out] print(out, mean(d), median(d)) # 120 15.9 4.5 | 500 53.9 4.5 | 1000 103.9 4.5
python — library equivalent import numpy as np a = np.array([2, 3, 3, 4, 4, 5, 5, 6, 7, 120]) a.mean(), np.median(a) # → (15.9, 4.5) — same answer
The mean of 15.9 is dragged up by a single 120-minute outage and is beaten by 90% of tickets — it makes the service look far worse than customers typically experience, and it's the wrong basis for an SLA that customers will hold you to. Propose reporting the median (4.5 min, the typical experience) alongside a high percentile such as p90 or p95 that explicitly captures the bad tail. SLAs are almost always written as percentiles precisely because a single mean can't express both "typical is fast" and "worst case is bad." A strong answer notes the right-skew is structural (resolution times are bounded below by 0 with a long upper tail), so median-plus-percentile is the honest default for any latency-like metric, and that you'd show the whole distribution (histogram or ECDF), not one number — which is where Plots That Tell the Truth picks up.
We just watched two definitions of "the middle" disagree violently. That raises the obvious next question: how many definitions of the middle are there, and how do you pick the right one? Chapter 1 defines all three — mean, median, and mode — precisely, and builds a rule for choosing.
Chapter 0 showed the mean and median disagreeing. Now we pin down all three measures of center precisely, prove why the median resists outliers, and build a rule for picking the right one for the decision in front of you.
Define each from zero, one sentence each, assuming nothing. Mean: add every value and divide by the count — the balance point of the data. Median: sort the values and take the middle one (or the average of the two middle ones when the count is even) — the value that splits the data into a lower half and an upper half. Mode: the value that occurs most often — the peak of the histogram.
Work the mean carefully, because "balance point" is the key intuition. Imagine placing each data value as an equal weight on a ruler; the mean is where the ruler balances. That's exactly why a far-out value tips the balance — it sits on a long lever arm. Physically motivating the mean this way makes its outlier-sensitivity obvious rather than a surprise.
Work the median operationally, so you can always compute it. Sort first (the single most-forgotten step). Then if n is odd, the median is the value at position (n+1)/2; if n is even, it's the average of positions n/2 and n/2 + 1. The median only ever looks at the middle rank, never at the magnitudes of the extremes — and that is the entire source of its robustness.
Prove the robustness claim with a thought experiment, not a formula. Take any dataset and push the largest value toward infinity. The mean goes to infinity with it, because that value is inside the sum. The median cannot move past the next value in from the edge — the middle rank is fixed, and the extreme just stays "the biggest," its rank unchanged. So the median has a bounded response to an outlier while the mean has an unbounded one.
Handle the mode honestly, quirks included. It's the only measure that works on non-numeric categories (the most common product, the most common error code). It can be multi-valued — bimodal data has two modes, foreshadowing Chapter 6. And for continuous data the "mode" is really the peak of a smoothed density, not any single observed value. Mode is underused for numbers and essential for categories.
Here is the decision rule the chapter is building toward. Symmetric, outlier-free data (test scores that cluster) → the mean is fine and uses all the information. Skewed or outlier-prone data (income, latency, wealth) → the median tells the truth about the typical case. Categorical or "what's most common" questions (most-popular SKU, most-frequent failure) → the mode. And when the mean and median diverge a lot, that gap is itself a skew alarm to investigate, not paper over.
Connect the three with the classic skew relationship as a reading aid. For a right-skewed distribution — a long tail to the right, like income — the order along the axis is mode < median < mean, with the mean pulled furthest toward the tail. For left-skewed it reverses. So just knowing whether mean > median or mean < median tells you which way the distribution leans before you even plot it.
A histogram you sculpt. Three lines overlay it: MEAN, MEDIAN, MODE (the tallest bar). In a symmetric pile all three sit almost on top of each other; drag a long right tail out and the mean slides toward it while the median and mode lag — making the mode<median<mean ordering visible. The gaps toggle draws the mean−median distance as a labeled skew alarm.
Annual salaries (thousands of dollars): [38, 42, 45, 45, 52, 58, 200].
python — from scratch def mode(xs): counts = {} for x in xs: counts[x] = counts.get(x, 0) + 1 return max(counts, key=counts.get) # value with the highest count sal = [38, 42, 45, 45, 52, 58, 200] m, md = mean(sal), median(sal) print(m, md, mode(sal), "skew score:", m - md) # mean-minus-median = skew alarm print(mean(sal[:-1]), median(sal[:-1])) # drop the founder → 46.67 45.0
python — library equivalent import numpy as np, statistics a = np.array([38, 42, 45, 45, 52, 58, 200]) a.mean(), np.median(a), statistics.mode(a) # → (68.57..., 45.0, 45) — same
When the decision depends on the total, the mean is the right — and only — choice, because total = mean × count is an exact identity while the median has no such relationship to the sum. Concrete case: capacity or cost planning. To size a payroll budget you need mean salary times headcount, not median times headcount; to provision servers for total daily compute you need mean request cost times request count. The median would systematically under-budget on right-skewed data because it ignores the expensive tail that the mean (correctly, for totals) includes. A strong answer states the general rule — median for "typical member," mean for "aggregate/total" — and notes you often report both: the median for the experience story and the mean (or tail percentiles) for the capacity story.
The median turns out to be more than a special construction: it's simply the value with half the data below it — the 50th percentile. That single realization generalizes the median into an entire family of summaries (quartiles, p90, p99) that underlie every robust description of spread and shape. We build that machinery next.
Your latency SLA says "p99 < 200ms." Your kid's test score says "90th percentile." Both are percentiles — but what number is a percentile, exactly, and how do you compute one when the rank lands between two data points?
Start from the plain-English definition and never let go of it: the p-th percentile is the value such that p percent of the data lies at or below it. The 90th percentile of test scores is the score that 90% of test-takers scored at or under. The 99th percentile of latency is the response time that 99% of requests came in at or faster than. Everything else in this chapter is just how to compute that value precisely.
A quantile is the same idea on a 0-to-1 scale instead of 0-to-100: the 0.9 quantile equals the 90th percentile. "Percentile" and "quantile" are the same concept in different units, the way 50% and 0.5 are the same number — use percentiles for reports, quantiles in code and math.
Now the unifying move of the whole lesson: the median is the 50th percentile — half the data below, half above, exactly the definition from Chapter 1. The median was never special. It's one member of a continuous family, and once you can compute any percentile you can compute the median, the quartiles, and the SLA tail all with one procedure.
The quartiles are the three percentiles that cut the data into four equal-sized quarters: Q1 = 25th percentile (a quarter of the data below it), Q2 = 50th percentile = median, Q3 = 75th percentile (three quarters below it). Name them, because the box plot and the IQR of the next chapters are built entirely from Q1, Q2, Q3.
Here's the honest hard part almost every tutorial skips: what to do when the rank lands between two data points. State the convention explicitly, because hand-computation only matches software if you use the same rule. For a sorted list of n values indexed 0 to n−1, the p-th percentile sits at fractional position h = (n − 1) × p / 100 (this is numpy's default, called "linear"). If h is a whole number, read that value directly; if not, linearly interpolate between the two neighboring values by the fractional part of h.
Walk the interpolation so it's mechanical, not magical. Let h = (n−1) × p / 100, let k = floor(h) (the lower index) and f = h − k (the fraction past it). Then percentile = x[k] + f × (x[k+1] − x[k]). In words: start at the lower neighbor and move a fraction f of the way toward the upper neighbor. It's the exact same "walk f of the way from A to B" you'd use to blend two colors.
Flag the convention trap loudly. There are at least nine different percentile definitions in common software — numpy alone has "linear," "lower," "higher," "nearest," "midpoint"; Excel's PERCENTILE.INC differs from PERCENTILE.EXC; R has types 1–9. On small samples they give different numbers for the same percentile. The fix isn't to memorize all nine — it's to state which one you used and stay consistent. This lesson uses numpy's "linear" throughout so every hand number matches the code.
Why this matters operationally: percentiles are how every latency SLA, every "top 10%" cutoff, and every tail-risk threshold is actually defined. A p99 target is a promise about the 99th-percentile value. If your team computes p99 with a different convention than your monitoring tool, you'll argue about a real service being "in violation" when you simply disagreed on interpolation. Precision here prevents that fight.
A sorted row of data dots with rank ticks beneath. Drag the p slider; an animated pointer travels to fractional rank h = (n−1)·p/100, snaps between two dots, and draws the interpolation. Toggle the math to see the live h, k, f. Switch the convention to watch small-n percentiles disagree.
Sorted latencies (ms): [12, 15, 18, 20, 22, 25, 30, 40]. n = 8, so ranks run 0..7 and h = (n−1)·p/100 = 7·p/100.
python — from scratch def percentile(xs, p): s = sorted(xs); n = len(s) h = (n - 1) * p / 100 # fractional rank k = int(h); f = h - k # lower index + fraction past it if f == 0: return s[k] return s[k] + f * (s[k + 1] - s[k]) # walk f of the way to the next point lat = [12, 15, 18, 20, 22, 25, 30, 40] for p in (25, 50, 75): print(p, percentile(lat, p)) # 25 17.25 | 50 21.0 | 75 26.25
python — library equivalent import numpy as np np.percentile(lat, [25, 50, 75]) # method='linear' by default → [17.25, 21.0, 26.25] np.quantile(lat, [.25, .5, .75]) # the 0-1 form — identical values
The two systems almost certainly use different percentile conventions and/or different interpolation, which diverge most on the sparse tail where p99 lives — a single request can flip the number. Diagnose by pinning down each tool's definition (numpy "linear" vs nearest-rank vs the t-digest approximation many monitoring systems use for streaming percentiles), and whether they're computed over the same time window and the same request population (are health-check pings included?). Resolve by agreeing on one convention as the contractual definition of the SLA and computing both sides that way — and recognize that on a heavy tail, p99 is inherently noisy (a handful of points determine it), so the honest SLA might specify a window and an averaging method, not a single instantaneous percentile. Bonus signal: streaming estimators (t-digest, HDR histogram) trade exactness for memory and add their own small tail bias — a real source of these disputes at scale.
We now have one procedure that computes the median, the quartiles, and the SLA tail. The quartiles do double duty: they're also the raw material for a spread measure that survives outliers — the interquartile range. That's the next chapter, where we ask not just where the data sits but how far it scatters.
Two teams both average 50 story points a sprint. One delivers 48–52 every time; the other swings 10–90. Same center, wildly different reliability. The number that separates them is spread — and there are two kinds, one of which a single outlier can blow up tenfold.
A center alone can't distinguish a metronome from a rollercoaster. Two datasets with identical means can be tightly packed or wildly dispersed, and for latency, delivery, and error metrics the dispersion is often what you actually care about (a filter that's usually great but occasionally 5m off is worse than one that's steadily 1m off). Every honest summary is at least a (center, spread) pair.
Define the range first because it's the simplest: range = max − min. It's trivially computable and instantly ruined by a single outlier — it's literally defined by the two most extreme points. Useful as a quick sanity bound, useless as a robust spread. It sets up the need for something better.
Define the IQR (interquartile range) as the robust spread built from last chapter's quartiles: IQR = Q3 − Q1, the width of the middle 50% of the data. Because it's the distance between two percentiles and ignores everything outside them, an outlier in the tail moves it barely or not at all — it inherits the median's robustness. The IQR answers "how wide is the bulk of my data."
Build variance from first principles as "average squared distance from the mean." Step by step: take each point's distance from the mean (its deviation), square it (so negatives don't cancel positives, and big misses are penalized more), then average those squared deviations. Squaring is the key design choice — it makes variance sensitive to outliers (a far point has a huge squared deviation), the mirror image of the mean's sensitivity.
Define standard deviation as the square root of the variance, and explain why we bother with the root. Variance lives in squared units (squared milliseconds, squared dollars), which are meaningless to a human; the standard deviation is back in the original units, so "std = 3.5 points" is directly comparable to the data. Std is the everyday spread number; variance is its mathematically convenient inner form.
Address the n vs n−1 question head-on, because it confuses everyone. Dividing the squared deviations by n gives the population variance (you have every data point there is); dividing by n−1 gives the sample variance (your data is a sample and you're estimating a larger population's spread — the n−1, Bessel's correction, compensates for using the sample's own mean). numpy defaults to n (ddof=0); pandas defaults to n−1. State which you use; on small n they differ visibly.
Now the headline comparison the chapter is built around — std vs IQR under an outlier. Take a clean symmetric set, compute its std and its IQR, then replace one value with a wild outlier and recompute both. The std explodes because the outlier's squared deviation dominates the average; the IQR barely twitches because the outlier sits in the tail, outside the middle 50%. Same robustness lesson as center, now for spread: squaring makes std fragile, percentiles make IQR robust.
Seven data points with two spread bars: a STD bar (mean ± 1 std) and a IQR bar (Q1 to Q3). Drag the rightmost point out into the tail: the STD bar stretches to keep up while the IQR bar stays almost fixed. The shaded squares show each point's squared deviation — watch the outlier's balloon. Toggle ddof to compare /n and /(n−1).
Part A — quiz scores [2, 4, 4, 6, 9] (n = 5). Part B — the robustness demo on a symmetric set.
python — from scratch def variance(xs, ddof=0): m = mean(xs); n = len(xs) return sum((x - m) ** 2 for x in xs) / (n - ddof) def std(xs, ddof=0): return variance(xs, ddof) ** 0.5 def iqr(xs): return percentile(xs, 75) - percentile(xs, 25) a = [2, 4, 4, 6, 9] print(max(a) - min(a), variance(a), variance(a, 1), std(a), std(a, 1), iqr(a)) clean = [20,22,24,26,28,30,32]; out = [20,22,24,26,28,30,120] print(std(clean, 1), iqr(clean), std(out, 1), iqr(out)) # 4.32 6 36.07 6
python — library equivalent import numpy as np a = np.array([2,4,4,6,9]) a.std() # population std (ddof=0 default) a.std(ddof=1) # sample std np.subtract(*np.percentile(a, [75, 25])) # IQR; or scipy.stats.iqr(a)
"Mean ± std" quietly assumes a symmetric, roughly-Gaussian shape — it implies the data lives in a symmetric band around the mean, which is false for right-skewed latency: the mean is already pulled up by the tail, and the std is inflated by the same tail, so "mean − std" can even be negative (a nonsensical negative latency), a dead giveaway the summary is wrong for the shape. Propose median + IQR for the typical spread, plus explicit tail percentiles (p95, p99) since for latency the tail is the product-relevant part that IQR deliberately ignores. The general principle: match the spread to the center and the shape — symmetric gets mean±std, skewed gets median/IQR/percentiles. A strong candidate also notes that reporting the whole distribution (histogram or ECDF) sidesteps the summary-choice problem, and that std still has its place downstream because most statistical machinery (CIs, z-scores) is built on variance — The Statistics of Evaluation picks that up.
We can now describe any dataset with a center and a spread. But two numbers still hide the shape — the humps, the gaps, the tail. Next we meet the one plot that shows the entire distribution with zero binning choices and lets you read off any percentile by eye: the ECDF.
A histogram's story changes completely when you change the bin width — same data, different picture. There's a summary of the whole distribution that has no knobs to fiddle, reads off every percentile by eye, and never lies about the tail: the ECDF.
Frame the CDF (cumulative distribution function) by the question it answers, running-total style. Pick any threshold x; the CDF at x is the fraction of the distribution at or below x. F(200ms) = 0.99 means 99% of requests come in at or under 200ms. It's a function — one output for every possible threshold — that sweeps from 0 (below all the data) up to 1 (above all the data), never decreasing.
Contrast the CDF with the PDF (probability density function) using the integral relationship in plain words. The PDF is the smooth curve whose height says how dense the data is near each value — it's what a histogram approximates. The CDF is its running total: the CDF at x is the accumulated area under the PDF up to x. So the PDF shows "where the data piles up" and the CDF shows "how much data we've accumulated by here." Two views of one distribution; the CDF is the PDF integrated.
Now define the ECDF (empirical CDF) — the assumption-free, data-only version. Sort your n data points; the ECDF is a staircase that starts at 0 and steps up by exactly 1/n at each data value (bigger steps for repeated values). At any x it equals (number of data points ≤ x) / n — literally the definition of a percentile, read forward. No bins, no bandwidth, no smoothing: the ECDF is the single most honest picture of a finite dataset.
Drive home why "no binning choices" matters. A histogram forces you to pick a bin width, and different widths can make the same data look unimodal or bimodal, smooth or spiky — the analyst's choice leaks into the conclusion. The ECDF has no such knob; two people plotting the ECDF of the same data get the identical staircase every time. It trades the histogram's intuitive shape for total reproducibility.
Show how to read percentiles straight off the ECDF, closing the loop with Chapter 2. To find the median, go to height 0.5 on the y-axis and read across to the x-value where the staircase crosses it — that x is the 50th percentile. Height 0.9 → p90, 0.99 → p99. The ECDF turns "compute a percentile" into "read a graph," and turns "compute a CDF value" into "read the graph the other way" (pick x on the bottom, read the fraction up the side).
Introduce the complementary CDF (survival function) for tails: CCDF(x) = 1 − CDF(x) = the fraction of data above x. For heavy-tailed data (latency, wealth, file sizes) the interesting action is in the far right where the ordinary CDF is flattening against 1 and you can't see anything; plotting the CCDF, often on a log y-axis, stretches that tail open so "the top 1% of requests" becomes readable. The CCDF is the tail's magnifying glass — we'll use it in Chapter 7.
Here's the intuition the sim below makes tactile: the same data can be viewed as a histogram (binned density, PDF-like) or as an ECDF (cumulative staircase). Toggling between them shows they're two encodings of one distribution — the histogram's tall bar is the ECDF's steep rise; the histogram's empty gap is the ECDF's flat plateau. Learning to see one in the other is the core fluency of reading distributions.
One dataset, two views. In histogram mode a bin-width slider visibly changes the shape (the story). In ECDF mode the staircase is fixed no matter what — drag the crosshair to read a cumulative fraction at any x, or hit a snap button to read a percentile off a height. Toggle CCDF to flip to the survival view.
Task completion times (seconds), sorted: [1, 2, 2, 3, 4, 4, 6, 10]. n = 8.
python — from scratch def ecdf(xs): s = sorted(xs); n = len(s) return [(v, (i + 1) / n) for i, v in enumerate(s)] # (value, cumulative fraction) def cdf_at(xs, x): return sum(1 for v in xs if v <= x) / len(xs) e = [1, 2, 2, 3, 4, 4, 6, 10] for x in (2, 4, 6): print(x, cdf_at(e, x), 1 - cdf_at(e, x)) # CDF and survival at the tail
python — library equivalent import numpy as np xs = np.sort(data) ys = np.arange(1, len(xs) + 1) / len(xs) # plot(xs, ys) is the ECDF # or: scipy.stats.ecdf(data).cdf.evaluate(x); survival = 1 - ys
Reach for the ECDF whenever exact percentiles or tail behavior matter, or when you want a reproducible, choice-free picture. Off an ECDF a stakeholder can read any percentile directly (go to the y-height, read across) — the median, p90, p99, all from one plot — and can answer "what fraction is under our SLA threshold?" by reading up from an x-value; a histogram makes both eyeball-estimates at best. The ECDF also never hides a gap or a plateau the way a coarse histogram bin can, and comparing two ECDFs on the same axes shows dominance (one distribution entirely faster than another) at a glance. The histogram wins for communicating overall shape and modes to a general audience (a staircase is less intuitive than bars), so the strong answer is: ECDF for percentile/tail/threshold questions and reproducibility, histogram (or violin) for shape — and often show both. Bonus: the complementary CDF on a log axis is the right tool specifically when the tail is the story.
The ECDF shows everything but demands you read a staircase. Sometimes you need to compare dozens of distributions on one screen at a glance — and for that, five numbers and a fence rule squeeze a whole distribution into a matchbox. That's the box plot, next.
You have 200 datasets to compare on one screen. You can't plot 200 histograms. The box plot squeezes each distribution into five numbers and a fence rule that automatically flags the outliers — a whole distribution in a matchbox.
Introduce the five-number summary as the box plot's entire content: minimum, first quartile (Q1), median, third quartile (Q3), maximum — five numbers that between them pin down the center, the spread of the middle half, and the extremes. Everything the box plot draws is one of these five, plus the outlier rule. It's the quartiles of Chapter 2 given a visual body.
Map the five numbers onto the drawing so you can read any box plot forever after. The box spans Q1 to Q3 (so the box is the IQR, the middle 50% of the data), the line inside the box is the median (Q2), and the whiskers extend outward toward the extremes. The box's length is the interquartile range; a fat box means a widely-spread middle, a thin box a tightly-packed one.
Now the part everyone half-remembers wrong — where do the whiskers stop? Not at the true min and max. The standard (Tukey) rule sets a fence: lower fence = Q1 − 1.5×IQR, upper fence = Q3 + 1.5×IQR.
The whiskers extend only to the most extreme data point still inside the fences. Any point beyond a fence is drawn as an individual dot and labeled an outlier. So the whiskers show the range of the "normal" data and the dots isolate the anomalies.
Justify the magic 1.5 without hand-waving: it's a convention chosen so that, for roughly-normal data, only about 0.7% of points fall outside the fences — rare enough that flagged points are genuinely unusual, common enough that the rule isn't paranoid. It's not a law of nature (some tools let you tune it, and 3×IQR marks "far outliers"), but 1.5 is the near-universal default, so state it when you report.
Walk the outlier-detection logic as an algorithm you can run by hand: compute Q1, Q3, IQR; compute the two fences; any point below the lower fence or above the upper fence is an outlier and gets its own dot; the whiskers then reach to the largest/smallest non-outlier values. This is a fully mechanical, reproducible outlier rule — no judgment call, no eyeballing — which is exactly why it's baked into every plotting library.
Name the box plot's superpower and its blind spot in one breath. Superpower: density of comparison — you can line up dozens of box plots side by side (per group, per model, per day) and compare centers, spreads, and outliers across all of them at a glance, which no histogram grid can match. Blind spot (the Chapter 6 hook): the box plot shows only five numbers, so it is blind to the shape between them — it cannot tell a single hump from two humps if they share the same quartiles.
Here's the reading checklist for any box plot in the wild: median position inside the box (off-center = skew — a median hugging Q1 means a right tail); box length (middle-half spread = IQR); whisker asymmetry (a long upper whisker = right skew); and the outlier dots (how many, how far). Four glances and you've extracted the center, spread, skew, and anomalies of a distribution from a matchbox-sized picture.
Ten data dots and a live box plot. The box spans Q1–Q3, the median line sits inside, the fences (Q1−1.5·IQR, Q3+1.5·IQR) are faint dashed lines, and the whiskers reach to the last in-fence points. Drag the top dot slowly past the upper fence and watch it "pop" into a red outlier at the exact 1.5·IQR crossing. Toggle 1.5× vs 3× to see the multiplier change what counts.
Build times (minutes): [4, 7, 8, 9, 10, 11, 12, 13, 15, 40]. n = 10, so h = (10−1)·p/100 = 9·p/100.
python — from scratch def box_summary(xs): q1, med, q3 = [percentile(xs, p) for p in (25, 50, 75)] iqr = q3 - q1 lf, uf = q1 - 1.5 * iqr, q3 + 1.5 * iqr outliers = [x for x in xs if x < lf or x > uf] inl = [x for x in xs if lf <= x <= uf] return dict(q1=q1, med=med, q3=q3, iqr=iqr, lower_whisker=min(inl), upper_whisker=max(inl), outliers=outliers) print(box_summary([4,7,8,9,10,11,12,13,15,40])) # q1=8.25 med=10.5 q3=12.75 iqr=4.5 whiskers=[4,15] outliers=[40]
python — library equivalent import matplotlib.pyplot as plt plt.boxplot(build_times) # draws the Tukey box plot; numbers via np.percentile + the 1.5*IQR fence
Read the box plot's structure: the box gives the typical spread and the whisker at 300ms says "normal requests in this region top out around 300ms," while the three dots at 2s/5s/12s are points beyond the 1.5×IQR fence — genuine anomalies, not part of the routine spread. The action split matters: the box/whisker tells you the baseline experience (fine or not), and the outlier dots are individual incidents to investigate, not average away — a 12-second request is a specific failing call worth tracing, and outliers are often where the real bugs live. A strong answer cautions that box plots hide multiplicity and shape (you can't see how many requests cluster where, or whether latency is bimodal — a fast cache-hit mode and a slow cache-miss mode would look like one box), so you'd follow up with an ECDF or violin and with the actual p95/p99 for the SLA; and that with large n, box plots can flag many "outliers" that are just the natural tail, so the 1.5×IQR dots are a starting flag, not a verdict.
The box plot's superpower — five numbers — is also its trap. Because it knows only five numbers, two completely different-shaped distributions can produce the identical box. The next chapter shows exactly how it lies, and the plot that fixes it: the violin.
Here's an unsettling fact: two datasets can have the identical five-number summary — same min, quartiles, median, max — yet one is a single hump and the other is two separate clusters with an empty middle. The box plot draws them the same. The violin plot doesn't.
State the box plot's fatal blind spot precisely: it summarizes with five numbers, and infinitely many differently-shaped distributions can share those five numbers. In particular, a single central hump and a two-cluster (bimodal) distribution can have the same min, Q1, median, Q3, and max — so the box plot renders them identically while they tell completely opposite stories about your data.
Make the danger concrete before the fix. Imagine task completion times that are actually two populations — a fast group (cache hits) around 12 and a slow group (cache misses) around 42, with nobody in between. The box plot's median lands in the empty gap at ~27, a value no task ever took, and the box spans a middle 50% that's mostly hollow. The single most important feature of the data — that it's two distinct regimes — is completely invisible.
Introduce the violin plot as the shape-restoring fix: it draws a smooth density curve (a KDE — kernel density estimate) mirrored on both sides of a center line, so the width of the "violin" at any value shows how much data piles up there. A single hump makes a violin bulging in the middle; a bimodal distribution makes a violin with two bulges and a pinched waist — the empty gap becomes a visible neck.
Explain the KDE at an intuitive level, no formulas. To build the density curve, place a small smooth bump (a kernel) on top of each data point and add them all up. Where points cluster, the bumps overlap and pile into a tall peak; where points are sparse, the curve sags. The result is a smoothed histogram with no hard bin edges — one continuous curve tracing where the data lives.
Name the KDE's one knob honestly, because it's the violin's version of the histogram's bin width: the bandwidth (how wide each bump is). Too wide and you over-smooth — two real humps blur into one, hiding the very multimodality you came to find. Too narrow and you under-smooth — random noise sprouts fake spurious bumps. The violin is more informative than a box plot but reintroduces a smoothing choice the ECDF (Ch4) didn't have; report or sanity-check the bandwidth.
Position the violin against its cousins so you know when to reach for each. Box plot: maximal density of comparison, robust five-number backbone, but shape-blind — best for comparing many groups when you trust unimodality. Violin: shows full shape and catches multimodality, at the cost of a bandwidth choice and more ink per group. ECDF: shape-faithful and choice-free, but less immediately intuitive to a general audience. Many tools draw a box plot inside the violin — best of both: robust quartiles plus honest shape.
Give the practical rule: whenever you're about to trust a single "typical value" (a mean, a median, a box) for a metric, first check for multimodality — because a summary of a bimodal distribution describes a population that may not exist (the average of a bimodal set often lands in its empty valley). A violin or an ECDF is a five-second multimodality check that can save you from optimizing toward a nonexistent "typical user."
A box plot and a violin, side by side, on the same data. Drag the separation slider to pull one central cluster apart into two. As separation grows the violin develops a pinched waist and two bulges — while the box plot barely changes (its five numbers stay similar), dramatizing that the box is blind to the split the violin exposes. The bandwidth slider shows over/under-smoothing; the median marker lands in the violin's empty waist.
Fast cache-hits [10,11,12,12,13] and slow cache-misses [40,41,42,42,43], combined into [10,11,12,12,13,40,41,42,42,43]. n = 10, h = 9·p/100.
python — from scratch (a hand KDE) import math def kde(xs, grid, bw): out = [] for g in grid: dens = sum(math.exp(-0.5 * ((g - x) / bw) ** 2) for x in xs) out.append(dens / (len(xs) * bw * math.sqrt(2 * math.pi))) return out d = [10,11,12,12,13,40,41,42,42,43] grid = range(5, 50) dens = kde(d, grid, bw=3) # two peaks near 12 and 42, a valley between print(sum(1 for x in d if 24 <= x <= 28)) # 0 — the empty waist under the median
python — library equivalent import seaborn as sns sns.violinplot(data=d) # draws the KDE-based violin # or scipy.stats.gaussian_kde(d) for the density; np.percentile for the box numbers
Run a multimodality check: plot a violin (or, choice-free, an ECDF) per cohort. The likely finding is bimodality that the box plot masks — e.g. a "bounce" population that leaves in seconds and an "engaged" population that stays minutes, with almost nobody in between. That matters enormously: the median or mean session duration lands in the empty valley and describes a nonexistent typical user, so any product decision optimizing "the average session" is optimizing toward a phantom. The right move is to segment — treat the two modes as two populations and analyze each (why do bouncers bounce? what retains the engaged?) rather than reporting one center. A strong answer notes the diagnostic ladder — box plot for quick comparison, but confirm shape with violin/ECDF before trusting any central summary — and flags the KDE bandwidth caveat: an over-smoothed violin can itself hide the two modes, so cross-check with the ECDF, which has no bandwidth knob.
Bimodality is one way a summary lies: the shape has two humps and the center falls between them. The other big way is skew and heavy tails — where even a single-hump distribution makes the mean untrustworthy because a few rare giants dominate it. That's next, anchored on a robot that failed exactly once.
Ten robot missions run with tiny errors — 0.30m, 0.32m, 0.28m... — and one mission goes badly wrong at 4.10m. The average error is 0.69m, more than double what a typical mission achieves. One failure hijacked the headline number. This is the heavy-tail trap, and it's everywhere.
Define the shapes. A distribution is skewed when it's lopsided — a right-skewed distribution has a long tail stretching to the right (large values) with data crammed to the left. A heavy-tailed distribution takes this to the extreme: rare but very large values that are far more common than a bell curve would predict — the outages, the megabuck salaries, the catastrophic failures. Latency, income, file sizes, and robot error are all classically right-skewed and often heavy-tailed.
Show mechanically why the mean is dominated by the tail. The mean is the average of every value, so a single value that's 10× the others contributes 10× as much to the sum. In a heavy-tailed set the handful of extreme points carry a disproportionate share of the total, dragging the mean up toward them and away from where the bulk of the data sits. The mean isn't broken — it's faithfully reporting the tail, which is exactly the wrong thing when you want the typical case.
Run the estimation tie-in as the anchor example. Ten per-run robot RMSEs (meters): [0.30, 0.32, 0.28, 0.35, 0.31, 0.29, 0.33, 0.30, 0.34, 4.10]. Nine tight missions around 0.3m and one 4.10m blow-up. Mean = 0.692m and median = 0.315m. The mean is 2.2× the median; a report quoting "average error 0.69m" makes the robot look twice as bad as it typically is, while a report quoting only the median (0.315m) hides the catastrophic failure entirely. Neither number alone is honest.
Bring the box-plot machinery from Chapter 5 to bear and watch it nail this: Q1 = 0.3000, Q3 = 0.3375, so IQR = 0.0375m — an incredibly tight middle 50%. The upper fence Q3 + 1.5×IQR = 0.3375 + 0.05625 = 0.3938m, and the whiskers span just [0.28, 0.35]. The 4.10 sits way past the fence, so the box plot draws a tiny box with a lone outlier dot far to the right — a picture that says "usually excellent, one bad failure" at a glance, which is the true story.
The ECDF (Chapter 4) tells the same truth a different way. Sort the errors and the ECDF rises steeply through 0.28–0.35 to reach 0.90 (nine of ten missions), then goes flat — a long plateau — all the way out to the single 4.10 failure at the top. Read it as "about 90% of missions come in under 0.35m, then nothing until one mission at 4.10m." This error-CDF is the standard way robotics and systems papers report error distributions precisely because it shows both the tight bulk and the rare tail honestly.
Use the tail-magnifier tools. The ordinary CDF flattens against 1.0 in the tail where you can't see the failure clearly; the complementary CDF (survival, 1 − CDF), often on a log-x axis, stretches the tail open so the 4.10 outlier and the plateau before it are legible. For heavy-tailed data, log-x plus survival is the default lens — it turns an invisible tail into a readable one, which is why latency dashboards plot p99/p99.9 on log axes.
State the honest summary recipe for heavy-tailed metrics, tying the whole lesson together: report the median (typical case, robust), report a tail percentile like p90/p95/p99 (the bad case you're on the hook for — here p90 = 0.725m, p95 = 2.412m, both dragged up by the lone failure, which is the point of a tail percentile), and show the distribution (box plot or error-CDF) so nothing hides. The mean can be reported too, but never alone and never as "typical." This trio — median, tail percentile, plot — is how you summarize a heavy tail without lying in either direction.
The robot-RMSE set in two synced panels. Left: a box plot — a tiny box around 0.30–0.34, whiskers to [0.28, 0.35], a lone red outlier dot far right, a MEAN needle out in the empty space and a MEDIAN needle inside the box. Right: the error-CDF rising to 0.9 by 0.35 then flat to the 4.10 step. Drag the failure slider and watch the mean chase while the median, box, and whiskers stay frozen; toggle log-x survival to open the tail.
Per-run robot RMSEs (meters): [0.30, 0.32, 0.28, 0.35, 0.31, 0.29, 0.33, 0.30, 0.34, 4.10]. n = 10.
python — from scratch def heavy_tail_report(xs): m, md = mean(xs), median(xs) q1, q3 = percentile(xs, 25), percentile(xs, 75) uf = q3 + 1.5 * (q3 - q1) outliers = [x for x in xs if x > uf] return dict(mean=m, median=md, ratio=m / md, # ratio = skew alarm p90=percentile(xs, 90), p95=percentile(xs, 95), outliers=outliers) r = [0.30,0.32,0.28,0.35,0.31,0.29,0.33,0.30,0.34,4.10] print(heavy_tail_report(r)) # mean 0.692, median 0.315, p90 0.725, p95 2.412, outliers [4.10]
python — library equivalent import numpy as np from scipy import stats np.median(r), np.percentile(r, [90, 95, 99]), stats.skew(r) # skew>0 confirms right skew
The single mean of 0.69m is the worst possible summary here: it's 2.2× the typical 0.315m median (so it makes a genuinely excellent stack look mediocre) and it buries the actual story — one catastrophic 4.10m failure — inside an average that reads as merely "okay." Propose the heavy-tail trio: median 0.315m (typical performance is excellent), the flagged outlier / a tail percentile like p95 ≈ 2.41m (there's a rare severe failure mode), and the full error distribution as a box plot (tiny box, one far outlier dot) or an error-CDF ("90% under 0.35m, one at 4.10m"). The failure is the finding, not noise to average away — you'd investigate that one run. A staff-level answer connects this to the downstream decision: for a safety spec you care about the tail (p99/worst case), not the average, so reporting only the mean is doubly wrong. This is exactly the discipline Evaluating Estimators, Fusion & Robots formalizes for filters and robots, and whether 10 runs is enough to trust any of these numbers is eval-statistics' sampling-error question.
We've now seen every way a single number can betray you: skew pulls the mean, outliers explode the std, bimodality strands the center, heavy tails dominate the average. Time to put them all on one screen at once and watch, in real time, which summaries lie under each distortion. That's the Distribution Detective.
One dataset, every summary live at once: histogram, ECDF, box plot, and the mean/median/percentile markers, all on screen. Now break the data — inject an outlier, split it into two modes — and watch in real time exactly which summaries lie and which survive.
This is the synthesis of the whole lesson. Instead of learning each summary in isolation, you watch all of them respond to the same data simultaneously. The histogram (shape), the ECDF (choice-free cumulative truth), the box plot (robust five numbers + outlier fence), and the mean/median/percentile markers — several views, one dataset, updating together as you distort it.
Start from the calm baseline so the distortions are legible: a clean, roughly-symmetric unimodal set. Here everything agrees — mean ≈ median (both near the center), the box is roughly symmetric with the median line centered, and the ECDF is a smooth S-curve. This agreement is the signature of well-behaved data: when every summary tells the same story, any one of them is trustworthy.
Distortion 1 — inject an outlier. The moment one point flies far out, the summaries disagree. The mean chases the outlier (Ch1); the median holds (Ch1). The std stretches (Ch3); the IQR doesn't (Ch3).
The box plot flags the point as an outlier dot beyond the fence (Ch5); the whisker stays put. The ECDF grows a flat plateau out to the lone step (Ch4). This divergence is diagnostic: when summaries disagree, you've found skew or an outlier.
Distortion 2 — split into two modes. Pull the data into two clusters. Now the box plot and the mean/median lie — the median lands in the empty valley between the modes (Ch6), the box spans a mostly-hollow middle. But the ECDF develops a visible double-step (a steep rise, a flat middle, a second steep rise). Multimodality breaks the centers and the box; it does not break the shape-faithful views.
Extract the general rule the detective teaches: match the failure mode to the summary that survives it. Outlier/skew → mean and std lie, median and IQR and the box-plot fence hold. Multimodality → mean, median, and box plot lie, the ECDF holds. The ECDF holds under both (it's the choice-free ground truth), which is why it's the summary to reach for when you don't yet know what's wrong with your data.
Turn it into a workflow you'll actually use: (1) glance at the ECDF or a shape view first to learn the shape — one hump? symmetric? heavy tail? two modes? (2) then pick the center/spread that fits — mean/std for clean symmetric, median/IQR for skewed, segment-and-report-separately for bimodal. (3) always show a tail percentile if the tail matters. The mistake the whole lesson has been fighting is picking the summary before looking at the shape.
One dataset in three synced views — histogram, box plot, and ECDF — with MEAN/MEDIAN/p90 markers threaded across them. A verdict strip reads AGREE (green) or DISAGREE (amber). Inject an outlier: mean/std/box-fence react, median/IQR hold. Split into two modes: the median strands itself, the ECDF double-steps. Reset returns to the agreeing baseline.
Clean [20,22,23,24,25,26,27,28,29,31]; inject outlier (31 → 200); or make it bimodal [10,11,12,12,13,40,41,42,42,43].
python — from scratch (the detective) def detective(xs): m, md = mean(xs), median(xs) near = sum(1 for x in xs if abs(x - md) <= (iqr(xs) or 1)) if abs(m - md) > 0.15 * (iqr(xs) or 1): return "centers DISAGREE → skew or outlier" if near == 0: return "centers agree but hollow middle → suspect bimodality, check the ECDF" return "centers agree AND data near center → well-behaved" print(detective([20,22,23,24,25,26,27,28,29,31])) # well-behaved print(detective([10,11,12,12,13,40,41,42,42,43])) # suspect bimodality
python — library equivalent import pandas as pd, numpy as np pd.Series(xs).describe() # count/mean/std/min/quartiles/max in one call np.percentile(xs, [90, 95, 99]) # add the tail; plus a violin/ECDF for shape
Look at the shape first, choose summaries second — the entire lesson's thesis. Concretely: (1) plot the ECDF and/or a violin to learn the shape — one hump or several? symmetric or skewed? heavy tail? — because the ECDF is choice-free (no bins, no bandwidth) and the fastest way to see multimodality and tails. (2) Based on the shape, pick the center and spread that fit: mean+std for clean symmetric unimodal, median+IQR for skewed or outlier-prone, and segment into separate populations if bimodal. (3) Always add tail percentiles (p90/p95/p99) when the tail carries the risk or the SLA. (4) Flag outliers with the 1.5×IQR fence and investigate them rather than averaging them away. The justification for the ordering is the whole lesson: picking a summary before seeing the shape is exactly how the mean lies on skewed data and how a box plot hides a second mode — shape-first prevents both. A strong close names the handoffs: whether these summaries are stable on a fresh sample is eval-statistics, and plotting them without deceiving is eval-plots.
You can now read any distribution honestly and predict which summary moves under which distortion. The last two chapters turn that into interview-ready reflexes, then file the whole toolkit into a one-screen cheat sheet and point to where each thread continues across the family.
Descriptive-stats questions are quietly lethal in interviews because they expose whether you actually understand data or just memorized formula names. Let's drill the reflexes that separate "I know what a percentile is" from "I know when the mean is lying."
The meta-skill interviewers probe: given any "the average X is …" claim, instinctively ask about the shape — is it skewed? outlier-prone? bimodal? — before trusting the number. Rehearse the 10-second triage: "skewed → mean lies, use median; tail matters → add a percentile; two modes → segment; and which convention computed that percentile?"
Reflex 1 — mean vs median for an SLA/metric. Hear "average latency 200ms" and immediately ask for the median and p99, because latency is right-skewed and the mean is pulled by the tail. The instant follow-up phrasing: "is that mean or median, and what's the p99?" — this single question signals you understand that averages hide tails, the #1 descriptive-stats interview tell.
Reflex 2 — why p99 not the mean for latency. A mean latency can look great while 1% of users suffer 10-second waits; those users churn. Tail percentiles (p95/p99/p99.9) are how you promise and measure the worst typical experience. Be ready to say "the mean is a capacity number; the percentiles are the experience number" out loud.
Reflex 3 — box vs violin. Box plot when you're comparing many groups and trust unimodality (dense, robust, five numbers); violin (or ECDF) when shape matters and you suspect multiple modes. Know the box plot's blind spot cold: it's shape-blind between its five numbers, so it can't see bimodality — the classic gotcha question.
Reflex 4 — ECDF vs histogram. Histogram for communicating shape to a general audience, but it has a bin-width knob that can lie; ECDF for reading exact percentiles, checking tails, comparing distributions, and total reproducibility (no bins). "Read any percentile straight off the ECDF" is the line that shows fluency.
Reflex 5 — summarize a heavy-tailed metric honestly. Median (typical) + tail percentile (the bad case) + the distribution (box/ECDF), never the mean alone. Quote the robot-error anchor if useful — "mean 0.69 vs median 0.315, one 4.10 failure: report both centers AND the outlier." Heavy tails always need a center and a tail.
Reflex 6 — the percentile interpolation gotcha. On small samples different conventions (numpy linear vs nearest vs Excel's two functions vs R's nine types) give different percentile values, so two correct tools can disagree on p99 — the fix is to state and pin the convention, not argue about which is "right." This subtle question catches people who only know percentiles superficially.
How to structure answers at staff level: state the shape framing first ("this is skewed, so the mean misleads"), do one crisp calculation or read-off, name the honest summary, then land on a decision or a recommended plot. Use the drill below like flashcards until "skewed → median + percentile" is automatic.
A flashcard drill. The card poses a scenario and a rapid question ("mean or median for the SLA?", "box or violin here?", "which summary lies?"). Tap your answer, hit Reveal for the correct one plus the one-line reason, and build a streak. Next card deals another.
"Our API's average latency is 180ms and we're under our 200ms SLA — ship it?" Sample: [40,50,55,60,60,65,70,80,90,1200] (ms).
python — the snippet you keep in your folder def summary_triage(xs): m, md = mean(xs), median(xs) rec = "right-skewed: report median + p95, not the mean" if m > 1.2 * md else "roughly symmetric: mean is fine" return dict(mean=m, median=md, ratio=m / md, p50=percentile(xs, 50), p90=percentile(xs, 90), p95=percentile(xs, 95), p99=percentile(xs, 99), recommend=rec) print(summary_triage([40,50,55,60,60,65,70,80,90,1200])) # mean 177, median 62.5, p90 201
python — library equivalent import numpy as np np.mean(x), np.median(x), np.percentile(x, [90, 95, 99]) # the four numbers behind every latency-SLA answer
The Q&A bank. Expand each for a staff-level model answer and a follow-up.
The mean of 15.9 is dragged up by a single 120-minute outage — nine of the ten tickets came in under 15.9, so the "average" describes almost nobody and makes the service look far worse than customers typically experience. The median is the average of the two middle sorted values (4 and 5) = 4.5 minutes, the honest typical experience. I'd report the median for the typical case AND an explicit tail percentile (p90/p95) for the bad case, because response times are structurally right-skewed (bounded below by zero, long upper tail) and no single number can express both "usually fast" and "occasionally an outage." This is exactly why SLAs are written as percentiles rather than means. Ideally I'd also show the distribution itself (histogram or ECDF) so the one outlier is visible rather than smeared into an average.
Follow-up: The PM says "but the mean uses all the data, isn't that more complete?" How do you respond?
The p-th percentile is the value below which p percent of the data falls — a rank position, not a distance along the value axis. To compute it (numpy's linear convention) I convert the percent to a fractional rank: for n = 8, the position is h = (n−1)·p/100 = 7·0.75 = 5.25. That means start at index k = 5 and move the fraction f = 0.25 toward index 6. The values there are x[5] = 25 and x[6] = 30, so p75 = x[5] + f·(x[6]−x[5]) = 25 + 0.25·(30−25) = 25 + 1.25 = 26.25. The key subtlety is that different tools use different conventions (numpy has five, Excel has two, R has nine), so on small samples they give different numbers — I always state which convention I used so results reproduce. The median is just the 50th percentile by the same procedure, which is what unifies the whole family.
Follow-up: Your monitoring tool and your offline script disagree on p99 by 15ms. Both are "correct." Why, and how do you resolve it?
Std is wrong when the data is skewed or has outliers, because squaring the deviations hands enormous weight to extreme points — a single outlier can inflate the std many-fold (I've seen 4.3 jump to 36 from swapping one value) even though the bulk of the data didn't change. It's also misleading whenever it implies a symmetric band around the mean that the data doesn't have: on right-skewed latency, "mean minus std" can be negative, a nonsensical negative latency that signals the summary doesn't fit the shape. The robust alternative is the IQR (Q3 minus Q1, the width of the middle 50%), which ignores the tail by construction and pairs naturally with the median and the box plot. General rule: match the spread to the center and the shape — mean+std for clean symmetric, median+IQR (plus tail percentiles) for skewed. Std still earns its place downstream because most statistical machinery is built on variance, but as a descriptive spread on real skewed data, the IQR is the honest one.
Follow-up: Why does most of statistics (confidence intervals, z-scores) build on variance rather than the IQR if the IQR is more robust?
Because the mean hides the tail, and the tail is what users actually suffer. Latency is heavily right-skewed — most requests are fast, a few are very slow (queueing, GC pauses, cache misses) — so a great-looking mean of 180ms can coexist with 1% of requests taking multiple seconds, and those users churn. The mean is a capacity number (total load / count); the percentiles are the experience number. p99 promises "the worst 1% of requests are still under X," a real, checkable commitment about the bad-but-not-rare case; p99.9 tightens it for critical paths. A single slow outlier barely moves p99 (it's determined by rank, not magnitude), which is exactly the robustness you want in an SLA. The tradeoff is that extreme-tail percentiles are noisy — a handful of points determine p99.9 — so at scale you compute them with streaming estimators (t-digest, HDR histogram) over a defined window, accepting small approximation error for bounded memory.
Follow-up: Your p99 is fine but users are complaining. What might a single percentile still be hiding?
They differ because a box plot summarizes with only five numbers (min-in-fence, Q1, median, Q3, max-in-fence) and is blind to the shape between them, while a violin draws the full density and can show multiple modes. The classic case: a bimodal distribution — two clusters with an empty middle — produces a box plot with a median stranded in the empty gap and a fat box spanning mostly-hollow space, whereas the violin clearly shows two separate bulges. I trust the violin (or, even better, the choice-free ECDF) for shape, because the box plot literally cannot represent multimodality. The box plot's strength is elsewhere: comparing many groups at once with robust five-number backbones, when I already trust unimodality. So the honest workflow is to check shape with a violin/ECDF first, then use box plots for dense comparison — and be aware the violin has its own knob (KDE bandwidth) that can over-smooth two real modes into one, which is why I cross-check with the ECDF.
Follow-up: Your violin looks unimodal but a colleague insists there are two populations. How do you settle it?
With a center AND a tail AND the distribution, never the mean alone. Here the mean is 0.692m and the median is 0.315m — the mean is 2.2× the median because a single 4.10m failure hijacked the average, so quoting the mean makes a genuinely excellent system look mediocre. But the median alone (0.315m) hides that catastrophic failure entirely. The honest summary is: median 0.315m (typical performance is excellent), a tail percentile like p95 ≈ 2.41m or the flagged 1.5×IQR outlier at 4.10m (there's a rare severe failure), and the distribution as a box plot (tiny box, one far outlier dot) or an error-CDF ("90% of runs under 0.35m, then a plateau to a lone 4.10m failure"). The failure is the finding — I'd investigate that one run, not average it away. And which summary I emphasize depends on the decision: for a safety spec I care about the tail (p99 / worst case), so the mean is doubly wrong there.
Follow-up: Only 10 runs produced these numbers. How much should that change your confidence, and whose problem is that?
Four traps. (1) Convention drift: numpy "linear," numpy "nearest," Excel's PERCENTILE.INC vs .EXC, and R's nine types give different values on small samples, so two "correct" tools disagree on p99 — the fix is to state and pin one convention as the contractual definition. (2) Rank-vs-distance confusion: treating the 90th percentile as "90% of the way from min to max" instead of "the value with 90% of the DATA POINTS below it" — badly wrong on skewed data. (3) Tail instability: extreme percentiles (p99.9) are determined by a handful of points, so they're noisy and shift run-to-run — report them over a window, and know streaming estimators (t-digest) add their own small tail bias. (4) Aggregating percentiles: you cannot average p99s across servers to get a global p99 — percentiles don't sum that way; you must merge the underlying distributions (or the sketches). Avoiding all four comes down to knowing that a percentile is a rank on a specific dataset with a specific convention, and being explicit about all three.
Follow-up: You have per-shard p99 latencies from 10 servers. How do you get the fleet-wide p99?
Shape first, summaries second. (1) Plot the ECDF and/or a violin to learn the shape — one hump or several? symmetric or skewed? heavy tail? — because the ECDF is choice-free (no bins, no bandwidth) and the fastest way to spot multimodality and tails. (2) Pick center and spread to fit: mean+std for clean symmetric unimodal, median+IQR for skewed or outlier-prone, and segment into separate populations if it's bimodal (a single summary of two modes describes a phantom that doesn't exist). (3) Add tail percentiles (p90/p95/p99) whenever the tail carries the risk or the SLA. (4) Flag outliers with the 1.5×IQR fence and investigate them rather than averaging them away. The ordering matters because picking a summary before seeing the shape is exactly how the mean lies on skewed data and how a box plot hides a second mode — shape-first structurally prevents both. I'd also note the limits: these describe the sample, not whether it's a stable estimate or the right population.
Follow-up: Your ECDF shows a clean single S-curve but the dataset is only 8 points. What do you actually know?
Wait times are right-skewed, so "average" is the most game-able framing on the slide. First: mean or median? The average could drop just because one long-wait outlier shrank while the typical wait (median) didn't move — I'd demand the median before believing a typical-experience improvement. Second: the tail — a lower mean can coexist with a worse p95/p99 if the change helped the middle at the expense of the worst cases, so I want the percentile customers actually suffer. Third: is it bimodal? If there are two populations (quick self-serve vs escalated cases), a single average is meaningless and the "win" might be a mix-shift between them rather than either group improving — check with a violin/ECDF and segment. Fourth: is 12→9 even stable or a small-sample lucky week (a sampling-error question)? My ask: "Show me median and p95 before/after, the ECDF for both periods, and a breakdown by case type." A 25% mean drop could be a real win, a mix shift, or one fewer outlier — only the shape-aware view distinguishes them.
Follow-up: They come back with medians that also improved 12→9. Are you convinced now?
The PDF (density) shows WHERE data piles up — its height at each value is the local density, and it's what a histogram approximates. The CDF is the PDF's running total: its value at x is the accumulated fraction of data at or below x, sweeping from 0 to 1 and never decreasing. So PDF answers "how dense here?" and CDF answers "how much accumulated by here?" — two views of one distribution, related by integration. The ECDF is the empirical, data-only CDF: a staircase stepping up 1/n at each observed value, equal to (count ≤ x)/n at any point. I reach for the ECDF over a histogram when I need exact percentiles (read any one straight off the y-axis), tail behavior, or "what fraction is under threshold X?", and whenever reproducibility matters — the histogram has a bin-width knob that can invent or erase features, while the ECDF is deterministic. The histogram (or violin) still wins for communicating overall shape and modes to a general audience, so I often show both; and for heavy tails I switch to the complementary CDF on a log axis to open the tail.
Follow-up: How would you use two ECDFs on the same axes to argue that model A is faster than model B everywhere?
Debate prompts. Each has two defensible positions and a middle path — be ready to argue any of them.
PRO: these metrics are all right-skewed, so the mean is systematically dragged by the tail and read as "typical" when it isn't — banning it forces the honest median+percentile framing and kills the most game-able summary on any slide. CON: the mean is the correct — and only — summary for totals and capacity (total payroll = mean × headcount; total compute = mean cost × count), which the median cannot give; banning it breaks legitimate budgeting, and the real fix is education (show median AND mean AND the distribution), not prohibition. MIDDLE: allow the mean only when explicitly labeled as an aggregate/capacity number and always shown beside the median and a tail percentile, never standing alone as "average X" where readers hear "typical."
PRO-VIOLIN/ECDF: box plots are shape-blind and routinely hide bimodality (median stranded in an empty valley), so on any data where shape is uncertain they can actively mislead; violins and ECDFs show the truth and modern tools make them free. PRO-BOX: box plots are unmatched for density of comparison — you can line up 50 groups and compare centers, spreads, and outliers at a glance, which a wall of violins can't match — and their robust five-number backbone plus the 1.5×IQR fence give a reproducible, knob-free outlier rule (unlike the violin's bandwidth choice). MIDDLE: check shape once with a violin/ECDF, then use box plots for dense multi-group comparison, or draw the box inside the violin to get robust quartiles and honest shape together.
PRO: with, say, 20 data points, p95 is determined by one or two values and swings wildly on a fresh sample — quoting it to two decimals implies a precision that doesn't exist, and different conventions even disagree on what it IS, so it's false confidence dressed as rigor. CON: the tail is often the entire point (the rare slow request, the catastrophic failure IS the risk you're managing), so refusing to report it hides exactly what matters — better to report the tail percentile WITH its uncertainty than to pretend the tail doesn't exist. MIDDLE: report small-sample tail percentiles only alongside an honest uncertainty range (a bootstrap/confidence interval — eval-statistics' tooling) and the raw outlier points, so the reader sees both the estimate and how little it's pinned down.
Reflexes drilled. The last chapter files every tool into a one-screen cheat sheet — quantity, definition, when it lies — and shows where each thread continues across the Evaluation & Analytics family this lesson is the foundation for.
You started with "15.9 minutes — real average or a lie?" You can now dismantle any one-number summary, read any distribution, and pick the honest description. Let's file the tools and point to where each thread continues.
Retell the lesson as one story: a single number can lie (Ch0) → choose the right center for the shape (Ch1) → the percentile machinery that generalizes the median (Ch2) → spread that survives outliers (Ch3) → the CDF/ECDF that shows the whole shape without binning (Ch4) → the box plot's robust five numbers and outlier fence (Ch5) → the violin that exposes the modes the box hides (Ch6) → heavy tails where even the mean of a unimodal set lies (Ch7) → and the detective that watches every summary agree and disagree at once (Ch8).
The one-screen cheat sheet — each quantity, its definition, and exactly when it lies:
| Quantity | Definition | When it lies |
|---|---|---|
| Mean | sum / n — the balance point | Skew or outliers pull it toward the tail; reports the tail, not the typical case |
| Median | the 50th percentile — half below, half above | Hides the tail entirely; strands in the empty gap of a bimodal set |
| Mode | the most frequent value | Multi-valued when bimodal; the only center for categories |
| Percentile p | value with p% at or below it; h = (n−1)·p/100 then interpolate | Convention-dependent on small n; rank not distance |
| IQR | Q3 − Q1 — width of the middle 50%, robust spread | Ignores the tail by design — blind to the failure the tail carries |
| Std | √(mean squared deviation) — spread in original units | Squaring makes it fragile to outliers; assumes symmetry |
| ECDF | fraction ≤ x — choice-free staircase | The honest ground truth — but less intuitive to a general audience |
| Box plot | five numbers + 1.5·IQR fence | Shape-blind between the five numbers — can't see two modes |
| Violin | KDE density — exposes modes | Bandwidth knob can over/under-smooth the modes away |
| Survival (CCDF) | 1 − CDF — the tail magnifier | Only useful when the tail is the story |
The decision tree in prose: symmetric & clean → mean + std. Skewed or outlier-prone → median + IQR + a tail percentile. Categorical or "most common" → mode. Suspect multiple modes → violin/ECDF, then segment. Need any exact percentile or a threshold fraction → ECDF. Comparing many groups at a glance → box plots (having first confirmed unimodality). The recurring rule: look at the shape (ECDF/violin) before you pick a summary.
Tap a branch that describes your data — symmetric-clean, skewed, categorical, multimodal, need-a-percentile, comparing-groups — and the tree lights the path to the right summary with its definition and the chapter to revisit. Reset clears the lit paths.
Clean symmetric set [1,2,3,4,5,6,7,8,9,10]. n = 10, h = 9·p/100.
Honest limitations of everything taught here: these are all descriptive summaries of the data in hand. They describe the sample; they do not tell you whether the sample is big enough to trust (a median of 10 points could shift a lot on a fresh 10), nor whether it represents the population you care about, nor how to plot any of it without deceiving. Descriptive statistics tells you what your data looks like; it cannot tell you it's the right data or a stable estimate.
Where each sibling continues the story:
| Lesson | How it builds on this foundation |
|---|---|
| Plots That Tell the Truth | Owns the PLOTTING of these summaries — drawing histograms, ECDFs, box and violin plots honestly (log axes, error bands) and detecting deceptive plots. This lesson owns what they mean. |
| The Statistics of Evaluation | Asks whether the medians and percentiles computed here are STABLE — standard errors, confidence intervals, hypothesis tests on these very summaries. This lesson computes them; that one quantifies their sampling error. |
| Evaluating Estimators, Fusion & Robots | Applies the median/percentile/error-CDF discipline to estimator and robot error distributions (filter RMSE, ATE/RPE, Monte-Carlo campaigns) — the heavy-tail robot-RMSE tie-in of Ch7 is its home turf. |
| Metric Design | Builds LOSS functions on these percentiles — quantile/pinball loss optimizes a model toward a chosen percentile, turning the descriptive quantiles here into training objectives. |
| Regression Testing & Quality Gates | Gates releases on latency percentiles (p95/p99) computed exactly as defined here, turning these summaries into CI thresholds and monitors. |
Latest in the field (2024–2026): p99/p99.9 latency SLOs remain the industry-standard way to specify service reliability (Google SRE practice, every major cloud's dashboards) — teams gate releases on tail percentiles, not the mean. Production LLM serving reports (vLLM, TensorRT-LLM) publish time-to-first-token and inter-token latency as median plus p90/p95/p99 because generation latency is strongly right-skewed (queueing, variable output length). Median household income is reported by national statistics agencies rather than the mean precisely because income is right-skewed. And streaming percentile estimators (t-digest, HDR Histogram, DDSketch) are standard in observability stacks (Datadog, Prometheus, OpenTelemetry) for computing p50/p95/p99 with bounded memory — the practical engineering answer to Chapter 2's "how do you even compute a percentile at scale."
Further reading, all foundational and honest: Tukey's Exploratory Data Analysis (1977) — the origin of the box plot, the 1.5×IQR fence, and the entire "look at the shape first" philosophy; the numpy/pandas percentile documentation for the interpolation conventions that decide your p99; and Hyndman & Fan (1996), "Sample Quantiles in Statistical Packages" — the paper cataloguing the nine quantile definitions and why software disagrees.
Never let one number speak for a distribution — always look at the shape first (ECDF or violin: one hump? skewed? heavy tail? two modes?), then pick summaries that fit. Symmetric and clean gets mean±std; skewed or outlier-prone gets median+IQR plus a tail percentile (p95/p99) since the tail is often the risk; categorical gets the mode; bimodal gets segmented into separate populations because one center describes a phantom. Flag outliers with the 1.5×IQR box-plot fence and investigate them rather than averaging them away. Report the convention for any percentile (numpy linear, etc.) so numbers reproduce. And know the limits: these describe the sample, not whether it's stable (confidence intervals) or well-plotted (honest visualization). The one-line reflex: the mean lies on skew, the median hides the tail, and the box plot can't see two modes — so look at the shape before you trust any summary.
"The greatest value of a picture is when it forces us to notice what we never expected to see." — John Tukey