Fisher Information &
the Cramér–Rao Bound

Chapter 1: The Likelihood — One Formula, Read Backwards

You already know the formula p(z; θ) — it says which measurements are probable once θ is fixed. Now the measurement has happened. Same formula, opposite reading: hold z fixed, sweep θ, and ask which hypothesis makes what-actually-happened least surprising.

Define the object carefully, because everything downstream differentiates it. The likelihood function is L(θ; z) = p(z; θ) viewed as a function of θ with the data z frozen. Insist on the asymmetry: over z (θ fixed) it integrates to 1 and is a probability density; over θ (z fixed) it integrates to whatever it wants and is not a density. It is a score sheet, not a belief. Beliefs over θ need a prior — that is Bayes Estimation's territory, and chapter 7 will reconnect.

Make it concrete immediately: one sample z = 4.2 from N(θ, 1). L(θ) is a Gaussian-shaped bump in θ centered at 4.2 — hypotheses near 4.2 explain the datum well; hypotheses far away rate it a freak event. Nothing new was computed. The formula was read in the other direction.

Independent samples multiply: L(θ) = ∏_i p(z_i; θ). Products of many small numbers are numerically vicious and analytically clumsy, so pass to the log-likelihood:

Logs turn independent evidence into addition — the single most consequential design choice in this lesson. Sums have means and variances, and chapter 3 lives on exactly that.

Work the Gaussian case to a shape you will meet fifty more times. For z₁…z_n from N(θ, σ²):

— an exact downward parabola in θ. Its peak sits where the derivative vanishes: at the sample mean. That peak location is the maximum likelihood estimate (MLE). You met MAP in Bayes Estimation; the MLE is MAP with a flat prior.

Worked Example: Three Samples, One Parabola

Setup. Three samples from N(θ, 1): z = (4.2, 3.6, 4.5). Build the log-likelihood, find its peak, and measure its shape.

Step 1 — write it.

Step 2 — peak. dℓ/dθ = (4.2−θ) + (3.6−θ) + (4.5−θ) = 12.3 − 3θ = 0, so the MLE is θ̂ = 4.1 — the sample mean.

Step 3 — shape. Expanding the squares, ℓ(θ) = −(3/2)(θ − 4.1)² + const — an exact parabola with second derivative −3 everywhere, which is −n/σ². For the Gaussian, the curvature does not even depend on the data values, only on the design (n, σ).

Step 4 — price a wrong hypothesis. ℓ(4.1) − ℓ(3.6) = (3/2)(0.5)² = 0.375 nats. Half a unit off the peak already costs 0.375 — and with n = 300 instead of 3 it would cost 37.5. The data's objection scales linearly with n.

Chapter 2: The Score — Every Data Point Votes

Stand at some hypothesis θ and ask each data point: which way should I move, and how urgently? That per-datum answer — the slope of its log-likelihood contribution — is the score, and the entire theory of estimation limits is bookkeeping on these votes.

Define the score:

— the sensitivity of a datum's log-plausibility to the hypothesis. Positive score: this datum argues θ should be larger. For the Gaussian, s = (z − θ)/σ²: the vote is the residual, scaled by confidence. A precise sensor (small σ) shouts; a sloppy one whispers.

Connect to the peak: the MLE is precisely where the votes sum to zero — the total score of the sample vanishes at θ̂. An estimate is a negotiated settlement among the data's demands; the sample mean is literally the θ at which the Gaussian residual-votes balance.

Now the central identity of the chapter: at the true θ, E[s] = 0. The proof is one honest line. The density integrates to 1 for every θ, so d/dθ of that integral is 0. Exchanging derivative and integral — this exchange is the regularity fine print; flag it now, it returns in chapter 4 — gives ∫(∂p/∂θ)dz = ∫s·p dz = E[s] = 0. Individual votes can be loud; at the truth they cancel on average.

Sit with what E[s] = 0 means operationally, because the sim makes it visceral. Draw data from the true θ and average their votes at the truth: the average shrivels toward zero as draws accumulate. Average the same votes at a wrong θ and they converge to a systematic nonzero pull toward the truth — (θ_true − θ)/σ² for the Gaussian. A persistent average vote is the data telling you that you are standing in the wrong place.

Do a non-Gaussian case by hand — the worked example below — because the Gaussian's linear score breeds bad intuition. For a Bernoulli(θ) bit, s(1) = 1/θ and s(0) = −1/(1−θ). At θ = 0.3, a success votes +3.33 and a failure votes −1.43: wildly asymmetric votes — yet they cancel exactly on average. The cancellation is a theorem, not a symmetry accident.

Worked Example: The Bernoulli Electorate

Setup. One Bernoulli(θ) bit with θ = 0.3: p(z; θ) = θ^z(1−θ)^1−z. Compute both possible votes, check they cancel on average, and compute their spread.

Step 1 — the score. log p = z log θ + (1−z) log(1−θ), so s = z/θ − (1−z)/(1−θ).

Step 2 — the two votes at θ = 0.3. A success gives s(1) = 1/0.3 = 3.333333 (pull θ up, hard); a failure gives s(0) = −1/0.7 = −1.428571 (pull down, gently). The asymmetry is real: successes are rare at θ = 0.3, so witnessing one is loud evidence.

Step 3 — cancellation at the truth.

Exactly zero — the theorem, not luck.

Step 4 — spread of the votes. Var(s) = E[s²] = 0.3 × (3.333333)² + 0.7 × (1.428571)² = 3.333333 + 1.428571 = 4.761905. Check the closed form: 1/(θ(1−θ)) = 1/0.21 = 4.761905. That spread is about to be named Fisher information in the next chapter — remember this number.

Chapter 3: Fisher Information — Variance of the Vote, Curvature of the Peak

Chapter 2 left a variance on the table. Chapter 1 left a curvature. Here is the punchline of classical estimation theory: they are the same quantity — and it is additive, so every new sample deposits a fixed amount of it in your account.

Define Fisher information as the variance of the score at the truth: I(θ) = Var(s) = E[s²] (the mean is zero by chapter 2). High information: each datum's vote is loud — small changes in θ change the log-plausibility a lot — the data discriminate. Low information: votes are murmurs; whole neighborhoods of θ explain the data equally.

Now the second face. Differentiate the identity E[s(θ)] = 0 with respect to θ once more (same regularity fine print). The product rule yields two terms — E[ds/dθ] + E[s²] = 0 — giving the information identity:

Variance of the slope equals expected downward curvature of the log-likelihood. The algebra is four lines; do each one on paper — it is quals question 1 verbatim.

Insist on the word expected, because this is the classic exam trap. The curvature of one realized log-likelihood is the observed information and jitters from dataset to dataset (for non-Gaussian models); Fisher information is its average over data drawn at the truth. For the Gaussian, the parabola's curvature happens to be data-independent (n/σ² exactly), which is precisely why Gaussians breed sloppy intuition here — the sim overlays thirty replicate log-likelihoods so you can see peaks jitter while the average curvature stands still.

Cash the additivity cheque. For iid samples, ℓ is a sum, scores add, and independent variances add:

Information is a budget: every sample deposits I₁. The Gaussian deposits 1/σ² per sample (n = 25 at σ = 2: total 6.25); the Bernoulli deposits 1/(θ(1−θ)) — chapter 2's 4.761905 at θ = 0.3, so a hundred coin flips hold 476.190476.

Give information its units and its warning label. I(θ) carries units of 1/θ² — its inverse is a variance in θ-units, which is exactly what makes chapter 4's bound dimensionally inevitable. And it is a local quantity, a property of the model at a parameter value: I(θ) can be large at one θ and tiny at another (Bernoulli near 0.5 versus near 0.01), so "how informative is my sensor" has no answer without saying where.

Worked Example: Both Faces, Then the Budget

Setup. Verify both faces and additivity with real numbers: Gaussian samples with σ = 2, then the Bernoulli budget from chapter 2.

Step 1 — face one (vote variance). The per-sample score is s = (z − θ)/σ², so Var(s) = σ²/σ⁴ = 1/σ² = 0.25 at σ = 2. A Monte Carlo of 200,000 scores confirms: 0.2506.

Step 2 — face two (curvature). d²/dθ² log p = −1/σ² = −0.25 per sample, so −E[curvature] = 0.25. The two faces agree, as the information identity demands.

Step 3 — additivity. n = 25 samples deposit I_total = 25 × 0.25 = 6.25. Preview of chapter 4: the best possible unbiased variance will be 1/6.25 = 0.16.

Step 4 — a non-Gaussian budget. The Bernoulli(0.3) deposit is 4.761905 per flip (chapter 2), so n = 100 flips hold I_total = 476.190476 — bounding any unbiased estimate of θ to variance at least 0.0021.

Chapter 4: The Cramér–Rao Bound — Cauchy–Schwarz in Four Moves

Here is the whole proof in one sentence: being unbiased handcuffs your estimator to the score with covariance exactly 1, and two random variables with a fixed covariance cannot both have small variance. Everything else is bookkeeping — four moves, no magic.

Move 1 — the handcuff. Unbiasedness says E[θ̂] = θ for every θ, not just one. That "for every" is the engine: a statement true along a continuum of θs can be differentiated in θ. Write E[θ̂] as ∫θ̂(z) p(z; θ) dz — the estimator is a fixed function of data; only p moves with θ.

Move 2 — differentiate the handcuff: d/dθ E[θ̂] = 1. Pushing the derivative through the integral — the same regularity fine print as chapters 2–3; third appearance, so name the conditions: support independent of θ, differentiable density, dominated derivatives — turns ∂p/∂θ into p times the score, giving E[θ̂ · s] = 1.

Move 3 — recenter. Since E[s] = 0, we get Cov(θ̂, s) = E[θ̂s] − E[θ̂]E[s] = 1. Pause on how strange this is: whatever unbiased estimator anyone builds — mean, median, a neural network, an algorithm from 2080 — its covariance with the score is exactly 1. Unbiasedness is not a loose promise; it is a rigid mechanical linkage to the data's votes.

Move 4 — Cauchy–Schwarz.

The estimator's identity appears nowhere on the right side — that is why this one line binds all algorithms at once, and why chapter 0's wall was real. With n iid samples, I = n I₁ and the floor is 1/(n I₁).

Read the equality condition out of Cauchy–Schwarz, because it is chapter 5's entire agenda: equality iff θ̂ − θ is exactly proportional to the score. For the Gaussian, θ̂_mean − θ = (σ²/n) × total score — proportional, so the sample mean sits on the floor at every n. The median is correlated with the score but not proportional to it: same covariance 1, extra orthogonal wobble, variance π/2 times the floor (worked below).

State the fine print as a first-class result, not a footnote: the bound requires the regularity conditions, and Uniform(0, θ) violates them — there the MLE (the sample maximum) has variance shrinking like 1/n², sailing straight through where a naive "1/nI" floor would sit. The CRLB is a theorem about smooth families, not a law of nature; the fine print is exam material and engineering material (quantized and thresholded sensors live near the edge of it).

Worked Example: The Floor, Who Touches It, and the Measured Handcuff

Setup. Gaussian location, σ = 2, n = 25. Compute the floor, check who touches it, and verify the covariance handcuff by Monte Carlo.

Step 1 — the floor. I_total = n/σ² = 25/4 = 6.25, so CRLB = 1/6.25 = 0.16.

Step 2 — the sample mean. Var(z̄) = σ²/n = 4/25 = 0.16 — on the floor, at finite n, because z̄ − θ = (σ²/n) × total score is proportional to the score (the equality case). MC confirms: Var 0.1599.

Step 3 — the handcuff, measured. Over 200,000 replications, Cov(z̄, total score) = 0.9991 — pinned to 1, as the proof demands of every unbiased estimator.

Step 4 — the sample median. Same covariance-1 handcuff, but not proportional to the score — its asymptotic variance is πσ²/(2n) = π × 4/50 = 0.251327, a factor π/2 = 1.570796 above the floor (efficiency 2/π = 0.63662). MC at n = 25 gives 0.246, within 2% of the asymptotic value.

Step 5 — read the two MC numbers together. Identical handcuff, different orthogonal wobble — the covariance constraint is what the bound prices; the wobble is what inefficiency looks like.

From Scratch: A CRLB Verification Harness

You should be able to check this whole chain numerically for any model you can sample and write a log-density for: E[score] = 0, the information identity by finite differences, and estimators raced against the floor.

python
import numpy as np

# CRLB verification harness, from scratch: for any model you can sample
# and log-density you can write, check the whole chain numerically:
#   E[score] = 0,  Var(score) = -E[curvature] = I(theta),  Var(est) >= 1/I.

def fisher_report(logpdf, sampler, theta, n, estimators, trials=100000, h=1e-4, seed=0):
    rng = np.random.default_rng(seed)
    Z = sampler(rng, theta, (trials, n))
    # score via central differences on the summed log-likelihood
    ll_p = logpdf(Z, theta + h).sum(axis=1)
    ll_m = logpdf(Z, theta - h).sum(axis=1)
    ll_0 = logpdf(Z, theta).sum(axis=1)
    score = (ll_p - ll_m) / (2 * h)
    curv  = (ll_p - 2 * ll_0 + ll_m) / h**2
    I_var, I_curv = score.var(), -curv.mean()
    print(f'E[score]      = {score.mean():+.4f}   (theory: 0)')
    print(f'Var(score)    = {I_var:.4f}')
    print(f'-E[curvature] = {I_curv:.4f}   (the two faces agree)')
    print(f'CRLB          = {1/I_var:.5f}')
    for name, est in estimators.items():
        v = est(Z).var()
        print(f'  Var({name:6s}) = {v:.5f}   efficiency = {(1/I_var)/v:.3f}')

# Gaussian location, sigma = 2, n = 25: mean is efficient, median pays pi/2
sig = 2.0
fisher_report(
    logpdf=lambda z, th: -0.5*np.log(2*np.pi*sig**2) - (z - th)**2/(2*sig**2),
    sampler=lambda rng, th, shape: rng.normal(th, sig, shape),
    theta=5.0, n=25,
    estimators={'mean': lambda Z: Z.mean(axis=1),
                'median': lambda Z: np.median(Z, axis=1)})

print()
# Bernoulli, theta = 0.3, n = 100: the sample proportion is efficient
th0 = 0.3
fisher_report(
    logpdf=lambda z, th: z*np.log(th) + (1 - z)*np.log(1 - th),
    sampler=lambda rng, th, shape: (rng.random(shape) < th).astype(float),
    theta=th0, n=100,
    estimators={'prop': lambda Z: Z.mean(axis=1)})

Chapter 5: Efficiency — Who Actually Hits the Floor

A floor is only interesting if someone can stand on it. Chapter 0's race showed the mean touching the floor under Gaussian noise and the median touching it under Laplace — same floor-touching honor, opposite estimators. Time to explain who gets to touch, when, and why the MLE almost always does eventually.

Formalize the scoreboard: the efficiency of an unbiased estimator is eff = CRLB / Var(θ̂), a number in (0, 1]; an estimator with eff = 1 is efficient. Chapter 4 already measured the Gaussian median at 2/π = 0.637 — now make that number systematic rather than anecdotal.

Derive the membership rule from Cauchy–Schwarz equality: θ̂ is efficient iff θ̂ − θ = c(θ) × s(θ; z) for some factor constant in z. Integrate this condition and you land on a structural theorem: a finite-n efficient unbiased estimator exists iff the family is an exponential family and θ is (a linear function of) its natural mean parameter, with θ̂ the corresponding sufficient statistic. Gaussian mean, Poisson rate, Bernoulli proportion: yes. Gaussian σ² with unknown mean, Laplace location, almost everything nonlinear: no finite-n winner exists at all.

Let the Laplace worked example land the punch: under Laplace(b = 1) noise the CRLB at n = 100 is 0.01. The sample mean — the reflex estimator — delivers 0.02: fifty percent efficiency, permanently, at every n. The sample median (which is the MLE here, since the log-density is −|z − θ|/b) closes on the floor as n grows. Matching the estimator to the noise family is worth a factor of 2 in variance — for free, forever.

Now the general engine — MLE asymptotics, sketched honestly in three steps you can follow on paper. (1) The MLE zeroes the total score: 0 = S_n(θ̂). (2) Taylor-expand around the truth: 0 ≈ S_n(θ) + S′_n(θ)(θ̂ − θ). (3) So:

The numerator tends to N(0, I₁) by the CLT (scores are iid, mean 0, variance I₁); the denominator tends to I₁ by the law of large numbers. The MLE is asymptotically efficient because it asymptotically becomes the proportional-to-score estimator that equality demands.

Connect back to the site's estimators to make this concrete: the Kalman filter under linear-Gaussian assumptions is the recursive MLE/MMSE and inherits efficiency (kalman-filter); robust estimators (Huber, from robust-estimation) deliberately spend efficiency at the Gaussian — about 5% at the usual tuning — to buy insurance against heavy tails. That lesson's tradeoff curve is this chapter's efficiency number turned into a dial.

Worked Example: Scoring the Reflex Estimator Under Laplace Noise

Setup. Laplace(b = 1) location noise (per-sample variance 2b² = 2), n = 100. Score the reflex estimator (mean) against the matched one (median = MLE).

Step 1 — Fisher information for Laplace location. log p = −log(2b) − |z − θ|/b, so s = sign(z − θ)/b and I₁ = E[s²] = 1/b² = 1. The CRLB at n = 100: 1/(n I₁) = 0.01.

Step 2 — the sample mean ignores the noise shape. Var = 2b²/n = 0.02. Efficiency = 0.01/0.02 = 0.5 — fifty percent, at every n, forever.

Step 3 — the sample median is the MLE (it minimizes ∑|z_i − θ|). Its asymptotic variance is 1/(4 f(0)² n) with f(0) = 1/(2b): 1/(4 × 0.25 × 100) = 0.01 — exactly the CRLB, asymptotically.

Step 4 — seeded Monte Carlo, 200,000 replications. Mean variance 0.02 (on its nose); median variance 0.0116 — 16% above the 0.01 floor at n = 100, the visible cost of "asymptotic"; rerun mentally at larger n and the gap closes like 1/n.

Step 5 — moral with numbers attached. Matching estimator to noise family is worth a factor 2 here; impatience (finite n) costs another 16% temporarily. The floor prices both.

Chapter 6: Many Parameters — the FIM, the Matrix Bound, and the Price of Nuisance

Real problems never have one unknown. You want the slope; the intercept tags along. You want position; the clock bias tags along. Here is the uncomfortable theorem: the parameters you do not care about still charge you variance on the ones you do — unless the information matrix says they are orthogonal.

Lift the machinery in one motion. The score becomes the gradient vector s = ∇_θℓ; the Fisher information matrix (FIM) is

(both faces survive, same proof shape). Diagonal entries: per-parameter loudness. Off-diagonals: vote entanglement — when data cannot tell a change in θ₁ from a change in θ₂, the off-diagonal grows.

State the matrix bound and immediately de-mystify what a matrix inequality means: Cov(θ̂) ≥ I⁻¹ means a^TCov a ≥ a^TI⁻¹a for every direction a — every linear functional of the parameters has its own scalar floor. Direction a = e_j reads off the practical corollary everyone uses: Var(θ̂_j) ≥ [I⁻¹]_jj. Note the trap in the notation: [I⁻¹]_jj, NOT 1/I_jj — the whole chapter lives in the gap between those two.

Make the gap concrete with the line fit: y_i = a x_i + b + Gaussian noise, x at (0, 1, 2, 3), σ = 1. The FIM is [[14, 6], [6, 4]] — that 6 is the entanglement: raising the slope and lowering the intercept produce nearly the same fitted line over right-shifted x. Invert: [I⁻¹]₁₁ = 0.2, while 1/I₁₁ = 0.0714 — knowing the intercept would make the slope 2.8 times easier. The nuisance parameter b, which you never wanted, taxed your slope by 180%.

Name the general law. With θ = (ψ, λ) (interest, nuisance), the effective information for ψ is the Schur complement:

— information minus what the nuisance can mimic. Verify on the numbers: 14 − 6²/4 = 5, and 1/5 = 0.2, matching the inverse's corner exactly. Equality with the naive 1/I₁₁ holds iff the off-diagonal vanishes: parameter orthogonality.

Turn the law into a design lever, because this is where estimation theory becomes engineering: center the regressors. Shift x to (−1.5, −0.5, 0.5, 1.5): now ∑x = 0, the FIM is diagonal, and the slope's floor is 1/5 = 0.2 with no inflation — same hardware, same noise, same four measurements, the tax legislated away by parametrization. (The slope floor itself is unchanged — 0.2 either way, since ∑(x − x̄)² = 5 is shift-invariant — what changed is that the intercept no longer costs anything extra, and the two estimates decorrelate.)

Foreshadow the showcase: in chapter 9 the FIM's eigen-structure becomes literal geometry — each sensor contributes a rank-one slab of information along its measurement direction, off-diagonals become ellipse tilt, and a nuisance-like degeneracy (two nearly parallel slabs) becomes an ellipse stretching to the horizon.

Worked Example: The Exact Price of Not Knowing the Intercept

Setup. Fit y_i = a x_i + b + noise (σ = 1) with x at (0, 1, 2, 3). How well can ANY unbiased method know the slope a — with and without knowing the intercept b?

Step 1 — assemble. Per-sample gradients are (x_i, 1), so the FIM is (1/σ²) [[∑x², ∑x], [∑x, n]] = [[14, 6], [6, 4]]. The off-diagonal 6 says slope and intercept votes are entangled on this right-shifted design.

Step 2 — invert (det = 14 × 4 − 36 = 20): I⁻¹ = [[0.2, −0.3], [−0.3, 0.7]]. Slope floor with b unknown: [I⁻¹]₁₁ = 0.2.

Step 3 — if b were known: floor = 1/I₁₁ = 1/14 = 0.071429. The nuisance inflation is 0.2/0.071429 = 2.8: not knowing the intercept makes the slope 2.8× harder — with identical data.

Step 4 — Schur complement check. Effective slope information = 14 − 6²/4 = 5.0, so the floor is 1/5 = 0.2. Same answer through the front door.

Step 5 — centered design x at (−1.5, −0.5, 0.5, 1.5): ∑x = 0 kills the off-diagonal; ∑(x − x̄)² = 5.0 is unchanged, so the slope floor stays 0.2 — but now 1/I₁₁ ALSO equals 0.2: orthogonal parameters, zero nuisance tax, and the intercept estimate no longer covaries with the slope.

Chapter 7: Bias and Priors — Bending the Floor

Chapter 4's proof had one confession in it: unbiasedness. Remove that assumption and the floor does not vanish — it bends. And once a prior enters, the bend has a direction: toward the prior, below the unbiased floor, priced by a beautiful generalization that the Kalman filter has been achieving all along without telling you.

Redo move 2 of chapter 4 with a biased estimator, E[θ̂] = θ + b(θ): differentiating now gives E[θ̂s] = 1 + b′(θ). The handcuff's length changed. Cauchy–Schwarz delivers the biased CRLB:

If b′ is negative — the estimator systematically leans toward some anchor — the variance floor drops below 1/I. Bias legally buys variance.

Make the purchase concrete with the shrinkage estimator θ̂ = 0.5 z̄ (Gaussian, σ = 2, n = 4, so I = 1 and the unbiased floor is 1.0): bias b(θ) = −0.5θ, b′ = −0.5, bent floor (0.5)²/1 = 0.25 — and Var(0.5 z̄) = 0.25 achieves it. A quarter of the unbiased floor! The catch: variance is not error. Its MSE is 0.25 + 0.25θ² — below the unbiased 1.0 for |θ| < √3, above it beyond. Shrinkage is a bet that θ is small; the bent bound prices the bet's variance, not its regret.

Draw the moral before you over-learn it: the biased bound shows the unbiased CRLB is not sacred — but it prices each bias function separately and says nothing about which bet to make. To rank bets you must average over θ — and the moment you put a distribution on θ, you have rejoined Bayes Estimation.

Enter the Bayesian CRLB (the Van Trees inequality): for ANY estimator — biased, nonlinear, whatever — the average MSE over the prior obeys

where J_prior = E[(d/dθ log π(θ))²] is the prior's own Fisher information about θ. Data information and prior information add — the same additivity as chapter 3, now across sources of knowledge rather than samples. No unbiasedness anywhere: averaging over the prior is what replaces it.

Run the linear-Gaussian case where everything is exact: prior N(0, P₀ = 1), n = 4 samples at σ = 2. Data information n/σ² = 1.0; prior information 1/P₀ = 1.0; J = 2.0; floor 0.5 — HALF the frequentist floor of 1.0. The posterior mean shrinks z̄ by data-info/total = 0.5 (it IS the 0.5 z̄ estimator above — the bet the prior justifies), and its Bayes MSE, by seeded Monte Carlo, is 0.5014: on the Van Trees floor. In linear-Gaussian problems the bound is achieved exactly, at finite n, by the MMSE estimator you built in Bayes Estimation.

Close the triangle of the three floors now standing: unbiased 1/I (binds unbiased estimators at a point), bent (1 + b′)²/I (binds one bias function at a point), Bayesian 1/J (binds EVERYONE on average over the prior). Each weakening of the claim buys generality; knowing which floor a paper is quoting is a quals reflex.

Worked Example: Pricing the Shrinkage Bet, Then Letting the Prior Settle It

Setup. Gaussian measurements, σ = 2, n = 4 (so data information = 1.0, unbiased floor = 1.0). A prior N(0, P₀ = 1) becomes available. Price the shrinkage bet and the Bayesian floor.

Step 1 — unbiased frequentist floor. I = n/σ² = 1.0, so Var ≥ 1.0 for any unbiased estimator.

Step 2 — the shrinkage bet θ̂ = 0.5 z̄. Bias b(θ) = −0.5θ, so b′ = −0.5 and the bent floor is (1 − 0.5)²/I = 0.25. Its variance: 0.25 × (σ²/n) = 0.25 — the bent floor, achieved. Below the unbiased floor by 4×, legally.

Step 3 — the bill for the bet at fixed θ. MSE(θ) = 0.25 + (0.5θ)². At θ = 2: 0.25 + 1.0 = 1.25 — WORSE than the unbiased 1.0. The bet pays off only for |θ| < √3.

Step 4 — the prior settles the bet. J = data info + prior info = 1.0 + 1/P₀ = 2.0, so the Van Trees floor on average MSE is 1/J = 0.5 — half the frequentist floor.

Step 5 — achieved. The posterior mean shrinks by data-info/total = 0.5 — exactly the c = 0.5 estimator, now justified — and seeded MC over 400,000 draws of (θ, data) from the joint gives Bayes MSE 0.5014: the floor, achieved to MC precision. Priors are not philosophical decoration; they are information, and they add.

Chapter 8: Sufficiency — Processing Can Only Lose Information

Your pipeline throws data away constantly: it averages, thresholds, bins, summarizes. Which of those steps cost you accuracy that no downstream algorithm can ever recover? There is an exact accounting — and it explains, at last, why the median pays chapter 4's mysterious π/2 tax.

State the theorem first because it is short and shocking: for ANY function T of the data,

Processing is a one-way valve for Fisher information — the data-processing inequality of estimation. The proof sketch is worth doing on the board: the score of T is the conditional expectation of the score of Z given T (one honest computation via the factorization of densities), and conditional expectation shrinks variance (law of total variance) — so Var(score_T) ≤ Var(score_Z). Equality iff the score of Z is already a function of T.

Define sufficiency through that equality: T is sufficient when p(z; θ) factorizes as g(T(z); θ) h(z) (the Fisher–Neyman factorization) — all θ-dependence flows through T — and then score_Z IS a function of T, so I_T = I_Z: lossless compression. For n Gaussian samples, the log-density's θ-terms collect into n z̄θ − nθ²/2 (over σ²): z̄ is sufficient. One number carries ALL 25 samples' information about the mean — averaging is free.

Now the median mystery dissolves: the sample median is NOT sufficient for the Gaussian mean — it forgets exactly where each datum sits, keeping only rank information near the center. Chapter 4 measured the price empirically: efficiency 2/π. This chapter derives where 2/π comes from: near the center the median is driven by SIGN information — and a sign is a 1-bit measurement.

Compute the 1-bit sensor exactly, because it is the workhorse of the chapter and of modern cheap-sensor design. A comparator reports only whether z exceeds threshold t. This is a Bernoulli observation with success probability 1 − Φ((t − θ)/σ), and its Fisher information (chain rule through chapter 2's Bernoulli) is

At the sweet spot t = θ: φ(0)²/(1/4) = 4/(2π) = 2/π ≈ 0.6366 of the analog sample's 1/σ². A perfectly placed comparator keeps 63.7% of the information — and that same 2/π is the median's efficiency, no coincidence: both live on signs around the center.

Move the threshold and watch the information bleed — the design lesson: at d = 1 (threshold one sigma off the truth), I_1bit = φ(1)²/(Φ(1)(1−Φ(1))) = 0.438629 — you keep 43.9%, not 63.7%. The floor from n = 100 one-bit sensors with a well-placed threshold is 1/(100 × 0.63662) = 0.015708 versus 0.01 from analog samples. Numbers like these budget real systems: how many comparators buy one ADC (π/2 ≈ 1.57), what a misplaced threshold costs (another 1.45× at one sigma), when binning telemetry is safe (k-bin histograms recover information rapidly as k grows).

Flag the boundary honestly: Fisher information's data-processing inequality is about estimating θ locally; Shannon's mutual-information DPI is a cousin, not the same theorem (the information theory workbook draws the family tree). Quals will ask you to keep them apart.

Worked Example: Pricing Three Compressions Exactly

Setup. Gaussian samples, σ = 1. Price three compressions of the data exactly: the mean, the median, and a 1-bit comparator at two threshold placements; then budget n = 100 comparators against n = 100 analog samples.

Step 1 — baseline. Each analog sample carries I = 1/σ² = 1.0.

Step 2 — sample mean. Sufficient (factorization collects all θ-terms into z̄), so it keeps 100% — compression for free.

Step 3 — 1-bit comparator at the ideal threshold t = θ. The bit is Bernoulli with p = 1/2 and sensitivity φ(0), giving I = φ(0)²/(1/4) = 2/π = 0.63662 — 63.7% kept. The sample median's asymptotic efficiency is the SAME 2/π = 0.63662: the median is sign-driven near the center, so chapter 4's tax is this chapter's quantization loss in disguise.

Step 4 — off-center threshold, d = 1 sigma. I = φ(1)²/(Φ(1)(1 − Φ(1))) = 0.241971²/(0.841345 × 0.158655) = 0.438629 — 43.9% kept. Threshold placement alone costs a further factor 1.45.

Step 5 — the budget. n = 100 well-placed comparators give a floor of 1/(100 × 0.63662) = 0.015708, versus 0.01 for 100 analog samples. Equivalently π/2 = 1.5708 comparators buy one analog sample — the exact exchange rate between a $0.05 comparator and an ADC channel.

Chapter 9: Showcase — the Sensor-Geometry CRLB Explorer

Everything so far priced WHAT you measure. The showcase prices WHERE you measure from. Drag two sensors around a target and watch the theory assemble in front of you: information adding as rank-one slabs, the bound inverting into an ellipse, and a Monte Carlo cloud of actual MLE estimates hugging that ellipse — geometry becoming accuracy in real time.

Set up the estimation problem cleanly: target at unknown 2D position p; sensor i at known position q_i measures either range r_i = |p − q_i| + noise(σ_r) or bearing (angle to target) + noise(σ_b). The measurement gradients are the whole story: the range gradient is the unit vector u_i from sensor to target (moving the target radially changes range one-for-one; transversely, not at all); the bearing gradient is the PERPENDICULAR unit vector scaled by 1/d_i (angles change fastest for close targets).

Assemble the FIM as a sum of rank-one slabs — the geometric reading of chapter 6:

each slab constraining exactly one direction. A range sensor is a vise clamping the radial axis; a bearing sensor is a vise clamping the transverse axis with grip 1/(σ_b²d²) that weakens with the SQUARE of distance. Cross-range accuracy from a bearing sensor floors at d × σ_b — at d = 10 with σ_b = 0.05 rad, transverse information 4.0 and a floor of 0.5 units of standard deviation, however good the estimator.

Work the two-range-sensor case to a closed form you can carry forever: unit vectors separated by angle γ, both σ_r = 1. The FIM is u_Au_A^T + u_Bu_B^T with det = sin²(γ) and trace 2, so

At γ = 90°: total variance floor 2.0 — each direction independently clamped. At 30°: 8.0 — a factor FOUR worse with IDENTICAL sensors, purely from geometry. As γ → 0 the two slabs become parallel, the FIM drops rank, and the transverse direction is unconstrained: the ellipse runs off the screen. This is GDOP (geometric dilution of precision) — the reason GPS wants satellites spread across the sky, not bunched.

Close the loop with achievability, because a bound without a cloud is decoration: draw noisy ranges, solve the MLE by Gauss–Newton (vnav-09's machinery — one step from truth suffices in the linearized regime), and scatter 200,000 estimates: seeded MC covariance trace 8.013 against the bound's 8.0. The ellipse is not a pessimist's abstraction; the MLE cloud FILLS it.

Worked Example: 90° vs. 30° in Full Matrix Detail

Setup. Two range sensors, σ_r = 1, observing a target. Compare separation angles 90 and 30 degrees; check the bearing-sensor scaling; confirm achievability by seeded Monte Carlo MLE.

Step 1 — geometry to unit vectors. Put u_A = (0, 1) (sensor due south of target) and u_B at 30° from it: (sin 30°, cos 30°) = (0.5, 0.866025).

Step 2 — assemble the slabs. FIM = u_Au_A^T + u_Bu_B^T = [[0.25, 0.433013], [0.433013, 1.75]]. det = 0.25 — and in general det = sin²(γ), so det at 30° = sin²(30°) = 0.25. Check.

Step 3 — invert. CRLB = [[7.0, −1.732051], [−1.732051, 1.0]], trace 8.0 — matching the closed form 2/sin²(γ) = 2/0.25 = 8.0. At γ = 90°: 2/1 = 2.0. Same two sensors, 4× the total variance floor, geometry alone.

Step 4 — ellipse anatomy. Eigenvalues of the CRLB are 4 ± 2√3 = 7.4641 and 0.5359 — the error ellipse is √(7.4641/0.5359) ≈ 3.73 times longer than wide, its long axis bisecting the two nearly-parallel sensor directions.

Step 5 — bearing scaling check. A bearing sensor at distance d = 10 with σ_b = 0.05 rad contributes transverse information 1/(σ_b²d²) = 4.0 — a cross-range standard-deviation floor of d × σ_b = 0.5.

Step 6 — achievability. 200,000 seeded MC trials, one Gauss–Newton step from truth on noisy ranges: empirical covariance trace 8.013 against the bound's 8.0 — the MLE cloud fills the ellipse; the bound is tight, not decorative.

Production Framing: A Sensor-Placement Evaluator

This is how a localization stack actually uses the CRLB: assemble the FIM for a candidate layout, score it by A/D-optimality, then cross-check that the covariance an NLLS solver reports at its optimum (inverse of the whitened J^TJ, GTSAM/Ceres-style) IS the inverse FIM.

python
import numpy as np
from scipy.optimize import least_squares

# Production framing: a sensor-placement evaluator, the way a localization
# stack actually uses the CRLB. Build the FIM for a candidate geometry,
# score it (A/D-optimality), and cross-check that the covariance an NLLS
# solver reports at the optimum IS the inverse FIM (J' Sigma^-1 J)^-1.

def fim_for_layout(target, sensors):
    """sensors: list of (pos, kind, sigma); kind in {'range','bearing'}."""
    I = np.zeros((2, 2))
    for pos, kind, s in sensors:
        d = target - pos
        r = np.linalg.norm(d)
        u = d / r                                  # unit vector sensor -> target
        if kind == 'range':
            g = u / s                              # d(range)/d(target) scaled
        else:                                      # bearing: transverse, 1/(s*r)
            g = np.array([-u[1], u[0]]) / (s * r)
        I += np.outer(g, g)                        # rank-one information slab
    return I

target = np.array([0.0, 0.0])
layout = [(np.array([0.0, -8.0]), 'range', 0.5),        # due south
          (np.array([-8.0, 0.0]), 'range', 0.5),        # due west: 90 deg baseline
          (np.array([10.0, 0.0]), 'bearing', 0.05)]     # bearing at d=10

I = fim_for_layout(target, layout)
C = np.linalg.inv(I)
print('FIM =\n', np.round(I, 4))
print('CRLB trace (A-optimality) =', round(float(np.trace(C)), 4))
print('CRLB det   (D-optimality) =', round(float(np.linalg.det(C)), 6))

# Cross-check: simulate one measurement set, solve the MLE by NLLS, and
# recover the covariance from the solver's Jacobian -- the observed FIM.
rng = np.random.default_rng(7)
def h(p):
    out = []
    for pos, kind, s in layout:
        d = p - pos
        out.append(np.linalg.norm(d) if kind == 'range' else np.arctan2(d[1], d[0]))
    return np.array(out)

sig = np.array([s for _, _, s in layout])
z = h(target) + rng.normal(0, sig)
res = least_squares(lambda p: (h(p) - z) / sig, x0=np.array([1.0, 1.0]))
J = res.jac                                       # already noise-whitened
C_solver = np.linalg.inv(J.T @ J)
print('solver covariance trace =', round(float(np.trace(C_solver)), 4))
print('matches CRLB at optimum:', np.allclose(np.trace(C_solver), np.trace(C), rtol=0.05))
# Placement loop in practice: sweep candidate layouts, keep the one with
# min trace(inv(FIM)) -- optimal experiment design, before any hardware.

Chapter 10: The Quals Arsenal

You have the machinery: likelihood, score, information, the bound, its equality case, its matrix form, its bent and Bayesian generalizations, and its geometry. Now sit the exam — eight questions of the kind that open an estimation quals, graded on derivations, limiting cases, and judgment rather than recall.

Attempt each question on paper (or the Teach whiteboard) before expanding the worked answer. The answers are graded sketches: full credit requires the algebra, the regularity flags where they belong, and a sanity check at a limit — precisely what the in-chapter derivations rehearsed. Every number in the answers is one the lesson has already verified, so checking your work means checking against chapters you can revisit.

The eight group by theme: foundations (the information identity with its fine print; the four-move proof and its failure model), structure (the (μ, σ²) FIM and attainability at finite n; the efficiency characterization), applications (nuisance/Schur on the line fit; the 1-bit quantizer design), and generalizations (Van Trees pricing the Kalman filter; the two-sensor geometry closed form).

Foundations

Structure

Applications

Generalizations

Debate Prompts — Oral-Exam Finishers

No single right answer exists, but there are defensible and indefensible positions — argue from the three-floors taxonomy, the regularity conditions, and the attainability results, not from vibes.

Chapter 11: Connections — Information as the Currency of Estimation

Strip the proofs away and this lesson installed a single accounting system: data deposit information, processing can only withdraw it, priors are deposits from elsewhere, and 1/I is the price of precision. Here is where that currency circulates across everything else you have built.

Every chapter was a ledger rule — deposits add (iid samples, independent sensors, priors), withdrawals are one-way (compression, quantization, nuisance parameters), and the CRLB is the conversion of balance into achievable variance. An estimate is a claim about your own ignorance; Fisher information is what makes the claim auditable.

Map the site, backwards first: bayes-estimation supplied the posterior machinery this lesson priced (the MMSE estimator ACHIEVES the Van Trees floor in linear-Gaussian problems); kalman-filter is that achievement run recursively — its P is the achieved Bayesian bound, and sf-09's information filter literally maintains Y = P⁻¹, this lesson's J, as its state. The "fusion is addition" theorem that distributed fusion agonizes over is Fisher additivity of independent sources, and its failure modes are double-counted information — the same currency, counterfeited.

Map forward into practice: vnav-09's Gauss–Newton covariance (J^TJ inverted at the optimum) IS the observed FIM inverted — every NLLS solver on the site has been quoting this lesson's bound without saying so; estimator-eval's chapter 4 races real filters against the CRLB, and its NEES machinery (with sf-21) detects filters whose reported covariance dips below what information arithmetic allows; sf-15's calibration states are chapter 6's nuisance parameters wearing hardware.

Map into design, where the FIM earns salaries: A/D/E-optimal experiment design (minimize trace, det, or max-eigenvalue of I⁻¹) is the showcase's sweep formalized — sensor placement, input-signal design for system identification, landmark selection in SLAM, satellite geometry (GDOP), and even active-learning acquisition functions are FIM shopping trips; the 1-bit threshold curve of chapter 8 is the same discipline applied to sensor front-ends.

Three doors this lesson deliberately did not walk through, with honest pointers: (1) information geometry — the FIM is a Riemannian metric on model space, Jeffreys's prior is √det I (you met improper priors in bayes-estimation), and NATURAL GRADIENT descent preconditions learning by I⁻¹, which is why K-FAC and friends cite Fisher; (2) misspecification — when the model is wrong, sandwich covariances replace 1/I (robust-estimation's territory); (3) Shannon's information — a cousin with its own data-processing inequality; the information theory workbook keeps the family tree straight.