Bayes' Rule
Updating Belief with Evidence

Chapter 1: Conditional Probability — Restricting the World to a Smaller Room

To fix the trap we need a tool, and the tool is older and simpler than Bayes' rule: conditional probability. It answers questions of the form "given that I already know X is true, how likely is Y?" Written P(Y | X) and read aloud as "the probability of Y given X." The vertical bar means "given" — everything to the right of it is something you already know for certain.

Here is the one mental image that makes conditional probability click, and it never leaves you: conditioning means restricting the world to a smaller room and re-measuring inside it. Normally you ask "what fraction of everyone is Y?" When you condition on X, you throw away everyone who isn't X, walk into the room containing only the X-people, and ask "what fraction of them is Y?" The denominator shrinks from "everyone" to "everyone who is X."

Let's make it concrete with a frequency picture — the single most powerful trick for taming probability. Forget percentages; imagine an actual crowd of 100 people standing in a field. We'll sort them by two yes/no facts and just count heads.

Suppose among these 100 people: 20 own a dog, and of those dog owners 15 also own a leash, while 5 own a dog but no leash. The other 80 own no dog (some own a stray leash, doesn't matter). Now ask: given that a randomly chosen person owns a dog, what's the probability they own a leash?

You do not divide by 100. You walk into the "dog owner" room — that's 20 people — and count leashes inside it: 15. So P(leash | dog) = 15/20 = 0.75. The denominator is 20, not 100, because conditioning on "dog" already discarded the 80 dogless people. That shrinking denominator is the whole idea.

Now the formula, which is just that counting procedure written with symbols. The fraction "leashes among dog owners" is (people with dog and leash) divided by (people with dog):

In our crowd P(dog and leash) = 15/100 = 0.15 and P(dog) = 20/100 = 0.20, so P(leash | dog) = 0.15 / 0.20 = 0.75. Identical to the head-count, because dividing both counts by 100 (to turn them into probabilities) doesn't change their ratio. The general statement, with any two events A and B, is the definition of conditional probability:

Two things to bolt down. First, P(A and B) is symmetric — "dog and leash" is the same group as "leash and dog" — but P(A | B) is not symmetric. P(leash | dog) = 0.75 here, while P(dog | leash) depends on how many leash owners there are total, which could be completely different. Conditioning is directional. That asymmetry is exactly what trips people in Chapter 0: P(positive | sick) is not P(sick | positive).

Second, rearrange the definition by multiplying both sides by P(B) and you get the multiplication rule, which we'll lean on in two chapters:

Read it as a recipe: to be in both groups, first be in B (probability P(B)), then within that room be in A (probability P(A | B)). 0.75 × 0.20 = 0.15 — the dog-and-leash fraction, recovered. Hold onto this; Bayes' rule falls right out of it.

One hand calculation to seal it. A deck of 52 cards: what's P(King | face card)? Face cards are J, Q, K — 12 of them. Kings among them: 4. Condition on "face card" means the denominator is 12, not 52. So P(King | face card) = 4/12 = 1/3. Compare P(King) unconditional = 4/52 = 1/13 ≈ 0.077. Knowing it's a face card raised the probability from about 8% to 33%, just by shrinking the room from 52 cards to 12.

Chapter 2: Bayes' Rule — Flipping the Conditional

We ended Chapter 1 with the directional problem: I can easily measure P(positive | sick) — just test a bunch of sick people and count — but what I actually want is P(sick | positive). The arrow points the wrong way. Bayes' rule is the machine that reverses the arrow. And it isn't a new axiom dropped from the sky; it falls out of the definition we already wrote, in three lines of algebra.

Start from the multiplication rule, written two ways for the same joint event. Using events Y (the hidden cause, e.g. "sick") and X (the observed evidence, e.g. "positive"):

The left sides are the same event — "X and Y" is the same crowd as "Y and X," order doesn't matter. So the right sides must be equal too:

Now divide both sides by P(X) to isolate the thing we want, P(Y | X), and we have Bayes' rule:

That's it. The entire lesson rests on this one line, and you just derived it yourself from "and is symmetric." Every piece has a name worth memorizing, because thinking in these names is what separates fluent reasoning from formula-plugging:

Read the rule as a sentence: your updated belief (posterior) equals how well the cause explains the evidence (likelihood) times what you believed before (prior), divided by how surprising the evidence was overall (evidence). Belief flows in, gets reweighted by how well each hypothesis predicted what you saw, and flows out re-normalized.

The evidence term P(X) usually isn't handed to you directly — you compute it by adding up every way the evidence could have happened. If Y is yes/no (sick / not sick), then X (positive) happens either via a sick person testing positive, or a healthy person testing positive:

This is the law of total probability: the total chance of the evidence is the sum of its chances under each hypothesis, each weighted by that hypothesis's prior. It's the denominator's full form, and notice the numerator of Bayes' rule, P(X | Y)·P(Y), is literally the first term of this sum. So the posterior is "the sick-and-positive piece, divided by the all-positive total" — exactly the 16-out-of-32 head-count from Chapter 0, now in symbols.

Quick hand calculation to anchor the names before the full worked problem. A bag has a fair coin (heads 50%) and a trick coin (heads 100%), picked at random. You pull one, flip it, get heads. P(trick | heads)? Prior P(trick) = 0.5. Likelihood P(heads | trick) = 1.0. Evidence P(heads) = 1.0·0.5 + 0.5·0.5 = 0.5 + 0.25 = 0.75. Posterior = (1.0 × 0.5) / 0.75 = 0.5 / 0.75 = 2/3 ≈ 0.667. One heads nudged you from 50% to 67% — not certainty, because a fair coin lands heads too.

Chapter 3: The Disease Test — Worked Fully, Every Step

Now we cash in. This is the exact problem from the opening, and we'll grind through every arithmetic step until the famous answer — 0.5, not 0.8 — is undeniable. The numbers come from a real assessment question, and they're chosen so the symmetry makes the punchline crisp.

The setup, stated precisely so we know which conditional is which:

Two new words, defined: sensitivity is the test's true-positive rate — of all sick people, what fraction does it correctly flag (0.80 here). Specificity is the true-negative rate — of all healthy people, what fraction does it correctly clear (also 0.80). "80% accurate" in the problem means both of these are 0.80. The false-positive rate is whatever's left over from specificity: 1 − 0.80 = 0.20.

We want the posterior P(D | +) — probability you're sick given a positive test. Bayes' rule:

Step 1 — the numerator (the sick-and-positive piece). Likelihood times prior:

Sixteen percent of all people are both sick and test-positive. In the 100-person crowd that's 16 people — the true positives.

Step 2 — the evidence P(+), the denominator, using the law of total probability. A positive can come two ways:

The first 0.16 is sick-and-positive (true positives); the second 0.16 is healthy-and-positive (false positives). They happen to be equal here, which is the heart of the punchline: the test produces exactly as many false alarms as true catches. In the crowd: 16 true positives + 16 false positives = 32 positives total.

Step 3 — divide:

Fifty percent. Of the 32 people who tested positive, only 16 are genuinely sick — a coin flip. The test was informative: it lifted your belief from the 20% prior up to 50%, a 2.5× jump. But "80% accurate" plus "positive" does not equal "80% sick." It equals 50%. The whole gap is the 16 false positives the base rate dragged in.

Play with the showcase grid above — it's the centerpiece. Set prevalence to 20, sensitivity to 80, specificity to 80 and read off 0.50. Now crank specificity to 99 (fewer false positives) and watch the posterior leap toward 95%: specificity, not sensitivity, is what a screening test most needs. Then drop prevalence to 1% and watch the posterior collapse even with a great test — that's Chapter 4 in one drag.

One more hand pass with different numbers, to prove it's the procedure not the magic 0.5. Suppose prevalence 0.10, sensitivity 0.90, specificity 0.90. Numerator = 0.90 × 0.10 = 0.09. Evidence = 0.09 + (0.10 × 0.90) = 0.09 + 0.09 = 0.18. Posterior = 0.09 / 0.18 = 0.50 again — because once more the true-positive and false-positive masses tie. The "coin flip" appears whenever sensitivity·prevalence equals false-positive-rate·(1−prevalence). Bayes' rule tells you exactly when your evidence is worth a coin flip and when it's worth more.

The grid above draws the four boxes for you. Now build the crowd yourself — assign all 196 people to their box from the prior, sensitivity, and specificity, then read the posterior straight off the picture by counting.

Chapter 4: The Base-Rate Effect — Why Screening Rare Things Is Brutal

We've nailed one case (prevalence 20%). The deeper lesson is what happens as the disease gets rarer. Hold the test fixed at 80% accurate and sweep the prevalence from common down to one-in-a-thousand. The posterior — the meaning of a positive test — doesn't drift gently. It collapses.

Let's hand-compute three points on that curve, all with sensitivity = specificity = 0.80, so the false-positive rate stays 0.20.

Prevalence 20% (1 in 5): numerator 0.80×0.20 = 0.16; evidence 0.16 + 0.20×0.80 = 0.32; posterior 0.16/0.32 = 0.500. (Our familiar coin flip.)

Prevalence 5% (1 in 20): numerator 0.80×0.05 = 0.040; false positives 0.20×0.95 = 0.190; evidence 0.040 + 0.190 = 0.230; posterior 0.040/0.230 ≈ 0.174. A positive now means only a 17% chance you're sick.

Prevalence 1% (1 in 100): numerator 0.80×0.01 = 0.0080; false positives 0.20×0.99 = 0.198; evidence 0.0080 + 0.198 = 0.206; posterior 0.0080/0.206 ≈ 0.039. A positive test, from an "80% accurate" test, means under 4% chance you're actually sick. You are 25× more likely to be a false alarm than a real case.

See the pattern? As prevalence shrinks, the numerator (true positives) shrinks with it, but the false-positive term barely moves — it's anchored to the enormous healthy population, which is now ~99% of everyone. The denominator becomes almost entirely false positives, and the true positives are a vanishing sliver of it. The posterior tracks true positives / (true + false positives), and when false positives dominate, that ratio crawls toward zero.

There's a beautiful way to feel why the curve falls so fast: the odds form of Bayes' rule. Define the likelihood ratio LR = sensitivity / false-positive-rate = 0.80 / 0.20 = 4. Then a positive test multiplies your odds (not probability) of being sick by 4:

At prevalence 20%, prior odds are 20:80 = 1:4 = 0.25. Posterior odds = 4 × 0.25 = 1, i.e. 1:1, which is probability 1/(1+1) = 0.50. At prevalence 1%, prior odds are 1:99 ≈ 0.0101. Posterior odds = 4 × 0.0101 ≈ 0.0404, probability 0.0404/1.0404 ≈ 0.039 — matching our long calculation. A positive test is a fixed 4× multiplier on the odds; if you start at terrible odds, 4× terrible is still terrible.

Misconception worth a callout, since it's the practical takeaway:

The sweep canvas plots the collapse as a curve. Now build the posterior that drives it, and watch a live town empty of sick people while the test keeps screaming positives — the screening paradox, frame by frame.

Chapter 5: Beyond — Chaining Evidence, Spam, and Classifiers

One positive test gave a coin flip. So what do you actually do? You collect more evidence — and Bayes' rule is built to chain. The key move is the engine of all rational learning: today's posterior becomes tomorrow's prior.

Take a second, independent test, also 80% accurate, and suppose it's also positive. After test one, your belief was 0.50 (that's your new prior). Run Bayes again with prior 0.50: numerator 0.80×0.50 = 0.40; evidence 0.40 + 0.20×0.50 = 0.40 + 0.10 = 0.50; posterior 0.40/0.50 = 0.80. Two independent positives carry you from 50% to 80%. In odds form it's effortless: each positive multiplies odds by LR = 4, so two positives multiply by 16. Prior odds 1:4 → 16 × 0.25 = 4 → probability 4/5 = 0.80. Same answer, one multiplication.

The same machine powers a naive Bayes classifier — the algorithm behind classic spam filters. The "hypothesis" Y is "this email is spam"; the "evidence" X is the words in it. For each word, you precompute a likelihood ratio (how much more often "viagra" appears in spam than in ham), and you multiply them all onto the prior odds. "Naive" means it pretends the words are independent — usually false, but it works shockingly well. A 12-word email is just twelve likelihood-ratio multiplications on the spam odds.

python
# Bayesian updating in odds form — chain any number of clues.
def update_odds(prior_odds, likelihood_ratio):
    return prior_odds * likelihood_ratio        # multiply, that's the whole update

def odds_to_prob(o):
    return o / (1 + o)

# Disease test: prevalence 20% -> prior odds 0.2/0.8 = 0.25; each + has LR = 0.8/0.2 = 4
odds = 0.25
for result in ["+", "+"]:        # two independent positive tests
    odds = update_odds(odds, 4.0)
print(odds_to_prob(odds))            # 0.8 — two positives -> 80%

# Naive Bayes spam filter: multiply a word's LR for every word seen.
def spam_prob(prior_odds, word_LRs):
    odds = prior_odds
    for lr in word_LRs:        # "free":8, "meeting":0.2, ...
        odds *= lr
    return odds_to_prob(odds)

And the direct probability-form version, computing the same disease posterior from scratch, so you can see they agree:

python
def posterior(prior, sens, spec):
    # P(disease | +) via Bayes' rule + law of total probability
    p_pos = sens * prior + (1 - spec) * (1 - prior)   # the evidence P(+)
    return sens * prior / p_pos

print(posterior(0.20, 0.80, 0.80))   # 0.5 — one positive
print(posterior(0.50, 0.80, 0.80))   # 0.8 — second positive (prior is now 0.5)

The canvas above chains one sequence you click out. Now build the odds update itself and run it on a whole cohort of patients at once — watch belief fan out, some climbing toward "sick," some sinking toward "healthy," each one just a string of multiplications.

Where this lesson plugs into the rest of the site:

• Bayes Filter — Bayes' rule run in a loop over time for a robot tracking its own position. Posterior-becomes-prior, every timestep.
• Kalman Filter — the same update, specialized to Gaussian beliefs and linear motion. The "trust slider" is Bayes' rule in disguise.
• Bayes Estimation — turning the posterior distribution into a single best estimate.
• Naive Bayes & GDA — the spam-filter classifier, derived in full.
• Hidden Markov Models — Bayesian updating when the hidden cause itself drifts over time.