Evaluating Generative Models
LLM · VLM · Diffusion

Chapter 1: Likelihood Is Not Quality

The most natural score for a generative model seems to sidestep the taste problem entirely. Instead of asking humans "which is better," ask the model itself: how probable does it find good, human-written data? A model that assigns high probability to good text must be a good model of language — right? So why does the lowest-perplexity sampler so often produce the dullest text?

Define the likelihood view in words. A language model is a probability distribution over sequences — given a context, it outputs a probability for every possible next token. Perplexity is the exponentiated average negative log-probability per token: intuitively, "the effective branching factor the model is choosing among" at each step. Perplexity 1 means it is certain; perplexity 50 means it is choosing as if among 50 equally likely tokens. (We keep this at intuition depth — the full mechanics of tokenizer normalization, sliding windows, and bits-per-byte belong to cs336-12-evaluation.)

Now the three blind spots, because each one is a way perplexity lies to you.

Blind spot 1 — likelihood is about the DATA, generation is about the MODEL's samples. Perplexity asks "how surprised is the model by human text?" It never asks "is the model's own output any good?" A model can score human text beautifully and still emit garbage under its own decoding settings. This is the likelihood trap: the single most-probable continuation is typically degenerate — repetitive, safe, dull — because real language spreads probability across many good continuations, and greedily grabbing the top one lands you in the boring valley between them.

Blind spot 2 — the fidelity-diversity tradeoff. A model that only ever emits one perfect poem has flawless fidelity and zero diversity. A model that emits wildly varied nonsense has the reverse. A single scalar — perplexity, or any per-sample quality score — cannot see the diversity axis at all. Plant the flag now: Chapter 5's precision/recall for generative models is the two-number instrument that finally measures both axes at once.

Blind spot 3 — memorization is the ultimate likelihood exploit. A model that regurgitates its training data assigns that data near-probability-1 and achieves near-perfect perplexity on that data, while having learned nothing generalizable. Evaluating on contaminated or memorized data is not measuring generation — it is measuring storage. This is why the 2023–2025 contamination findings (models scoring far above chance on "held-out" benchmarks whose items leaked into pretraining) forced the field toward fresh/private test sets like LiveBench's rolling questions.

Play the two axes before we compute anything. The simulation below shows the model's sample cloud against the true data distribution. Morph it from mode collapse (all samples in one tight blob — high fidelity, no diversity) through matched to hallucinating (samples spraying outside the data). Watch the two readouts move in opposition — and hit Memorize to see both scores hit a perfect, worthless 1.0.

Now work the arithmetic in both directions. A tiny model assigns these probabilities to the 5 tokens of a held-out sentence: 0.5, 0.25, 0.5, 0.125, 0.25. Compute perplexity by hand.

Step 1 — take natural logs of each probability.

Step 2 — sum, then average over the 5 tokens.

Step 3 — perplexity is exp of the negated mean.

Step 4 — sanity-check with powers of two, because these probabilities are all powers of ½. The product of the probabilities is 2⁻¹ × 2⁻² × 2⁻¹ × 2⁻³ × 2⁻² = 2⁻⁹, so perplexity is the geometric mean's reciprocal:

Same answer. The model behaves as if choosing among about 3.5 equally likely tokens at each step — that is what a perplexity of 3.5 means.

Step 5 — now the memorizer. A model that stored a training string assigns p = 0.99 to every token of it. Its perplexity on that string:

A near-perfect score — and utterly worthless as a signal of creative quality. Ask this model for a new poem and the metric has told you nothing. Likelihood on data it stored is storage, not generation.

python
# From scratch: accumulate log-probs in a loop, printing every intermediate
# so it mirrors the hand arithmetic exactly.
import math
def perplexity(probs):
    running = 0.0
    for p in probs:
        lp = math.log(p)
        running += lp
        print(f"  p={p:.3f}  log={lp:+.4f}  running={running:+.4f}")
    mean = running / len(probs)
    print(f"  mean log-prob = {mean:.4f}")
    return math.exp(-mean)

print("held-out:",  round(perplexity([0.5, 0.25, 0.5, 0.125, 0.25]), 4))  # 3.4822
print("memorized:", round(perplexity([0.99]*5), 4))                     # 1.0101

python
# Library: perplexity in one line.
import numpy as np
p = [0.5, 0.25, 0.5, 0.125, 0.25]
print(round(float(np.exp(-np.mean(np.log(p)))), 4))   # 3.4822
print(round(2**1.8, 4))                                 # 3.4822  (power-of-two check)

Two connections close the chapter. For diffusion and images, exact likelihoods exist only for some model families (autoregressive, normalizing flows); VAEs and diffusion give only a bound (the ELBO). Worse, likelihood in pixel space correlates notoriously poorly with visual quality — one more reason the field moved to sample-based distribution metrics (Chapter 5). And the chapter's law: likelihood metrics are cheap, differentiable, and excellent as a training signal and a regression alarm — but they are proxy metrics for generation quality. Every serious eval stack pairs them with sample-based and preference-based measurements.

Likelihood failed because it never looks at the model's own output. But sometimes generation does come with a partial answer key — code that must pass unit tests. There, quality per sample is binary again, and we can do far better than likelihood. Getting the estimator right takes one subtle piece of combinatorics, and the obvious approach gets it wrong. That is pass@k.

Chapter 2: pass@k, Derived — The Unbiased Estimator

Code generation hands us back an answer key: unit tests. Either the generated function passes the tests or it does not — a crisp checkmark. So evaluation is easy again, right? Only if you get one subtle piece of combinatorics right, and the natural approach gets it wrong in a way that has appeared in shipped papers.

Set the stage. For tasks with a verifier (code + unit tests, math with checkable answers), per-sample quality is binary. But generation is stochastic: one model call is one draw from the model's distribution, and calling it twice gives two different programs. Users care about a specific quantity: "if I let the model try k times and keep any attempt that passes, what is the chance at least one works?" That is pass@k. (Choosing pass@k as your metric is a definition decision — metric-design's territory; here we own its estimation.)

Derive the population quantity first. If each independent sample solves the problem with probability p, then all k failing has probability (1−p)^k, so:

Intuitively, k is a compute-for-reliability dial: with p moderate, pass@k climbs fast as you spend more attempts. A model that succeeds 30% of the time per try succeeds 1 − 0.7⁵ = 83% of the time given five tries.

The naive approach. Draw n samples, observe c successes, estimate p̂ = c/n, and plug in: 1 − (1 − c/n)^k. This looks unimpeachable — the right formula with the best estimate of p. It is biased, and here is exactly why. The function f(p) = 1 − (1−p)^k is strictly concave (it curves downward). By Jensen's inequality, the expected value of a concave function of a random estimate is less than the function of the true value:

The gap is largest exactly where you tend to operate — small n, k near n. We will see it hit 7 points at n=10, p=0.3, k=5.

Build the unbiased estimator from first principles. Given the n draws you actually made (c of them passing), imagine choosing k of them uniformly at random — that is itself a valid "run the model k times" experiment. The chance your chosen k contains no success is the number of failure-only subsets over all subsets:

So the estimator is one minus that:

Why it is unbiased, in one line of intuition. Each of the C(n,k) subsets of your n i.i.d. samples is a legitimate k-shot trial; averaging the all-fail indicator over all of them is an average of unbiased estimates, hence unbiased. (The scipy check below confirms E[estimator] = 1 − (1−p)^k to the digit.)

Play the bias reveal before the arithmetic. Draw ten samples from a hidden success rate p; the pass@k curve is drawn three ways — the true curve, the plug-in, and the unbiased estimator. Then hit Run 500 experiments and watch the plug-in curve sag below truth while the unbiased curve lands right on it.

The flagship worked example: n = 10 samples, c = 3 pass. Estimate pass@5.

Step 1 — count the failure-only subsets. There are n − c = 7 failures. The number of ways to pick 5 of them:

Step 2 — count all subsets of size 5 from the 10 samples.

Step 3 — the probability a random 5-subset contains no pass.

Step 4 — pass@5 is one minus that.

Step 5 — the numerically stable product form, which real implementations (HumanEval's reference code) use to dodge gigantic factorials:

Same value, factorial-free.

Step 6 — see the plug-in bias at the population level. Take the true rate p = 0.3, so true pass@5 = 1 − 0.7⁵ = 1 − 0.16807 = 0.83193. Average the plug-in estimator over c ~ Binomial(10, 0.3):

Boundary behaviors interviewers love. c = 0 gives pass@k = 0 exactly (no successes to catch). When n − c < k, we have C(n−c, k) = 0, so pass@k = 1 exactly — you cannot pick k samples without hitting a success. k = n returns the raw any-success indicator. And k = 1 gives c/n exactly — the plug-in is unbiased at k=1, because f is linear there.

python
# From scratch: pass@k three ways, all printing intermediates and agreeing.
from math import comb
import numpy as np

def pass_at_k(n, c, k):
    if n - c < k:                       # fewer failures than k -> guaranteed hit
        return 1.0
    combo = 1.0 - comb(n - c, k) / comb(n, k)          # explicit ratio
    prod  = 1.0 - np.prod(1.0 - k / np.arange(n - c + 1, n + 1))  # stable form
    print(f"  C({n}-{c},{k})={comb(n-c,k)}  C({n},{k})={comb(n,k)}")
    return combo, prod

print(pass_at_k(10, 3, 5))   # (0.91667, 0.91667)
print(pass_at_k(10, 3, 8))   # (1.0, 1.0)  -- boundary: only 7 failures

python
# Library: the official HumanEval estimator (Chen et al., 2021).
import numpy as np
n, c, k = 10, 3, 5
print(round(1.0 - np.prod(1.0 - k / np.arange(n - c + 1, n + 1)), 5))   # 0.91667

Practical estimation discipline. Choose n comfortably larger than the largest k you will report (HumanEval used n=200 for k up to 100). Report pass@1 alongside pass@k so readers can separate per-sample quality from sampling budget. And never compare models at different temperatures without saying so — temperature moves p per sample and the sample diversity that pass@k exploits. (How the harness executes untrusted code, sandboxing, and suite curation is ai-evaluation's job; contamination for these suites is cs336-12's; this chapter owns the estimator and its bias math.)

pass@k worked because a verifier gave us binary per-sample truth. But most generative quality has no verifier — only preferences: thousands of A-vs-B votes across many models. Votes are pairwise; a leaderboard is a single ordered list with error bars. Turning a tournament into a ranking is the next instrument.

Chapter 3: Pairwise Votes to Rankings — Bradley-Terry and Elo

You now have thousands of A-vs-B votes across ten models. But votes are pairwise, and your leaderboard must be a single ordered list. How do you turn a tournament into a ranking — with error bars?

Motivate the model with the flaw it fixes. Raw win rate against "whoever you happened to fight" is confounded by opponent strength: a model that only battled weak opponents looks great, a model that fought only champions looks mediocre. We need a latent-strength model that accounts for who you beat.

Bradley-Terry (1952) assigns each model a positive strength s_i, and models every battle as:

Equivalently, with ratings r_i = log s_i, this is P(i beats j) = σ(r_i − r_j), a sigmoid of the rating difference. Only differences matter, so we pin one model (or the sum of strengths) as an anchor — ratings are defined up to a shift.

Write the likelihood. For battle counts w_ij (times i beat j), the log-likelihood is Σ w_ij log(s_i/(s_i+s_j)). It is concave in the ratings r — there is one global optimum — and fitting it is a weighted logistic regression whose feature vector is +1 for model i, −1 for model j. That is exactly how modern LLM leaderboards (LMSYS / Chatbot Arena, since Dec 2023) fit their ratings: BT via logistic regression on the full battle log, not the online Elo update.

Fit it by hand with the classic MM (minorization-maximization) update:

where W_i is model i's total wins and n_ij the battles between i and j. From uniform strengths it converges to the maximum-likelihood fit. Let's watch it, then work the numbers.

The worked example: three models, 10 battles per pair. A beat B 7/10, A beat C 8/10, B beat C 6/10.

Step 1 — total wins. W_A = 7 + 8 = 15, W_B = 3 + 6 = 9, W_C = 2 + 4 = 6. Each n_ij = 10.

Step 2 — one MM iteration from uniform s = (⅓, ⅓, ⅓). Each pair's denominator term is 10 / (⅓ + ⅓) = 10 / (⅔) = 15, and each model faces two others, so each denominator = 15 + 15 = 30:

(Already normalized — they sum to 1.) One iteration already ranks them correctly.

Step 3 — iterate to convergence (the code does this in a loop):

Step 4 — read the fitted strengths through the BT lens. The converged model reconciles all three pairwise rates at once, smoothing each using all the data:

Step 5 — convert to leaderboard (Elo) units. The Chatbot-Arena scale is r_i = 400 log₁₀(s_i/s_ref), where a 400-point gap means 10:1 odds. Relative to the weakest model C:

An Elo-scale number is just a reparameterized BT strength.

Step 6 — the online Elo update, by hand. Elo is BT fit one battle at a time. The expected score is E_A = 1 / (1 + 10^{(R_B−R_A)/400}); after a result S ∈ {0, 0.5, 1}, R_A ← R_A + K(S − E_A). Take R_A = 1200, R_B = 1000, K = 32, and let A lose (an upset):

Ratings are zero-sum under equal K. K is a learning rate: big K tracks fast-changing players (fine for chess humans), but for fixed model checkpoints Elo's order-dependence is pure noise — which is why leaderboards refit BT offline on all battles.

Step 7 — the uncertainty reality check. The standard error of a single 7/10 win rate:

A 14.5-point standard error on a win percentage! Ten battles per pair order the medals but prove almost nothing. Rank confidence comes from bootstrapping over battles: resample the battle log with replacement, refit BT each time, report the 2.5–97.5 percentile band per model plus P(model i ranks first). Two models whose bands overlap are statistically tied, whatever the point estimates say. (The bootstrap machinery itself belongs to eval-statistics; here we apply it to battles.)

python
# From scratch: the MM loop, printing per-iteration strengths, then Elo utils.
import numpy as np
W = {('A','B'):7,('B','A'):3,('A','C'):8,('C','A'):2,('B','C'):6,('C','B'):4}
m = ['A','B','C']

def fit_bt(W, m, iters=2000):
    pi = np.ones(len(m)) / len(m)
    for _ in range(iters):
        new = np.zeros_like(pi)
        for i, a in enumerate(m):
            wins = sum(W.get((a,b), 0) for b in m if b != a)
            den  = sum((W.get((a,b),0)+W.get((b,a),0))/(pi[i]+pi[j])
                       for j, b in enumerate(m) if b != a)
            new[i] = wins / den
        pi = new / new.sum()
    return pi

pi = fit_bt(W, m)
print(pi.round(4), round(pi[0]/(pi[0]+pi[1]), 4))   # [0.5971 0.2453 0.1576] 0.7088
print((400*np.log10(pi/pi[2])).round(1))            # [231.4 76.8 0.]

def elo_update(Ra, Rb, score, K=32):
    Ea = 1 / (1 + 10**((Rb - Ra)/400))
    return Ea, Ra + K*(score - Ea), Rb + K*((1-score) - (1-Ea))
print(tuple(round(x, 2) for x in elo_update(1200, 1000, 0)))  # (0.76, 1175.69, 1024.31)

python
# Library: BT is logistic regression on +1/-1 model-indicator features over
# battles -- exactly how Chatbot Arena fits its leaderboard.
from sklearn.linear_model import LogisticRegression
# X row per battle: +1 in winner column, -1 in loser column; y = 1. Fit, then
# coefficients are the ratings (up to the anchor). Bootstrap rows for CIs.

We turned votes into a ranking with error bars — assuming the votes themselves are honest. Human votes cost dollars and days. A judge model costs cents and seconds. But your new measuring instrument has opinions of its own, and before you trust it you calibrate it — like any lab instrument. That is the next chapter.

Chapter 4: LLM-as-Judge, Measured — Position, Verbosity, Self-Preference

Human votes cost dollars and days; a judge model costs cents and seconds. The catch: your new measuring instrument has opinions of its own. Before you trust it, you calibrate it — exactly as you would a thermometer or a scale.

Frame the shift. LLM-as-judge replaces the human rater in the Chapter 3 pipeline. Everything downstream — BT fitting, leaderboards, bootstrap CIs — is unchanged. What changes is the error model of the vote itself. The MT-Bench / LLM-as-judge work (Zheng et al., NeurIPS 2023) showed that strong judges agree with humans ~80%+ of the time — comparable to human-human agreement — but only after specific biases are handled. This chapter is how to measure those biases. (Where judges sit in the harness is ai-evaluation's territory.)

Bias 1 — position bias, and its calibration experiment. Judge every pair twice, with positions swapped. Define per-pair outcomes: consistent-A (A wins both orders), consistent-B, and flip (the verdict follows position). The flip rate is your instrument's position sensitivity. The position effect is half the gap between "A wins when first" and "A wins when second." The correction: aggregate over both orders (equivalently, randomize per query) — the +b and −b terms cancel by symmetry, exactly like Chapter 0.

Bias 2 — verbosity bias, and its experiment. Take pairs where a known-equal (or known-worse) answer is padded — restatements, bullet expansions, no new content — and measure the win-rate lift per 100 added tokens. The cleanest causal design: the same answer at two lengths; the judge should be indifferent, so any systematic preference is the bias. Corrections: rubric instructions that demote padding, length-stratified reporting, or post-hoc length-controlled modeling (regress judge preference on length difference, read the quality coefficient at zero length gap — the idea behind Length-Controlled AlpacaEval, 2024).

Bias 3 — self-preference, and its experiment. Have judge J score outputs from J's own model family versus others, with authorship hidden; compare J's own-family preference against a panel of other judges (or humans) on the same pairs. The self-preference score is the gap between J's own-model win rate and the panel consensus. Mitigations: never judge your own family for headline numbers; use judge juries of different families; report per-judge results.

Formalize with a simple measurement-error model:

Each calibration experiment isolates one term by design: swap isolates position; same-content-two-lengths isolates γ; authorship blinding isolates the self term. This is instrument calibration, not vibes.

Run all three calibrations on the bench below. Set hidden bias dials, run the swap / pad / blind experiments, and watch each measured bias recover the planted dial value — the whole point of a designed experiment.

The worked example: judge 100 response pairs twice each, swapping positions. When A is first, A wins 60/100; when A is second, A wins 44/100. Per-pair breakdown: A wins both orders 40; A wins only-when-first 20; A wins only-when-second 4; B wins both orders 36.

Step 1 — consistency-check the breakdown. A-first wins = 40 + 20 = 60 ✓. A-second wins = 40 + 4 = 44 ✓. Total pairs = 40 + 20 + 4 + 36 = 100 ✓.

Step 2 — flip rate. The verdict flipped on the 20 + 4 discordant pairs:

Step 3 — position effect. Half the gap between the two orders:

Step 4 — the debiased win rate, averaging over both orders so the position term cancels:

Step 5 — is 52% actually above 50%? Treat the 200 judgments as independent for a first cut:

After debiasing there is no significant preference. The raw "60% win rate" headline was manufactured entirely by seating order. (The rigorous version pairs the two orders of each pair and tests only the 24 discordant pairs — a McNemar-style test, flagged for eval-statistics.)

Step 6 — the instrument verdict. A 24% flip rate fails a <15% acceptance bar. This judge needs a stronger rubric or both-order aggregation baked in before its votes feed any leaderboard.

Rubric-based judging reduces both variance and bias. Instead of "which is better?", score named criteria (correctness, groundedness, formatting), each with anchored 1–5 descriptions, then combine per a declared weighting (the weighting itself belongs to metric-design). Rubrics shrink the judge's freedom to indulge biases and raise agreement — but a rubric can encode the wrong values, so validate rubric scores against a human-labeled slice.

When is a judge "good enough"? Operational criterion: judge-human agreement on a held-out human-labeled set ≥ human-human agreement on the same set, and flip rate below your threshold (e.g. <15%), and no significant self-preference on the model families being compared. Below that bar, use the judge as a filter (triage obvious cases) and send the rest to humans.

python
# From scratch: the swap calibrator, printing every tally.
def calibrate_position(a_first_wins, a_second_wins, flip_only_first,
                       flip_only_second, n=100):
    flip_rate = (flip_only_first + flip_only_second) / n
    pos_effect = (a_first_wins/n - a_second_wins/n) / 2
    debiased  = (a_first_wins/n + a_second_wins/n) / 2
    print(f"  flips={flip_only_first+flip_only_second}  "
          f"first={a_first_wins}  second={a_second_wins}")
    return flip_rate, pos_effect, debiased

print(calibrate_position(60, 44, 20, 4))   # (0.24, 0.08, 0.52)

import math
se = math.sqrt(0.52*0.48/200)
print(round(se, 4), round((0.52-0.50)/se, 3))   # 0.0353 0.566

python
# Library: order-symmetrized win rate and flip rate over vote arrays.
import numpy as np
# v_first[i]  = 1 if judge picked A when A shown first for pair i
# v_second[i] = 1 if judge picked A when A shown second for pair i
debiased = (v_first.mean() + v_second.mean()) / 2
flip_rate = (v_first != v_second).mean()

Judges gave us cheap, calibratable votes for text. But an image has no preference vote at scale — just two clouds of samples, real and generated. How far apart are two clouds? That question has an exact classical answer, and it runs the entire diffusion literature. Next chapter.

Chapter 5: Image and Diffusion Metrics — FID From Scratch

For images there is not even a preference vote to lean on at scale — just two clouds of samples, real and generated. How far apart are two clouds? That question has an exact classical answer, and it runs the diffusion literature.

Set up the distribution-matching view. A generative image model is good if its sample distribution matches the data distribution — and this single object captures both fidelity and diversity (Chapter 1's two axes reappear). We cannot compare raw pixel clouds meaningfully (two pixels apart is not two "units" of quality), so first embed every image with a fixed feature network — classically Inception-v3's 2048-dimensional pool layer — then compare the feature clouds.

The Gaussian simplification. Fit a Gaussian N(μ, Σ) to each cloud — just the empirical mean vector and covariance matrix. The Fréchet distance (the 2-Wasserstein distance) between two Gaussians has a closed form:

Read each term in words. The mean term asks "are the clouds centered on the same place?" The trace term asks "do they have the same spread and shape?" — and the matrix square root is what correctly accounts for correlated dimensions.

Demystify the matrix square root. For diagonal covariances the matrix square root is just the elementwise square root of the diagonal. The correlated case needs scipy.linalg.sqrtm. An implementation gotcha: sqrtm of a near-singular product can return tiny imaginary parts — production FID code takes the real part.

Play with FID's behavior before computing. Move the model cloud's mean and reshape its covariance; the FID readout decomposes into its two terms, computed every frame from the actual closed form. Hit Mode collapse and watch the trace term explode while the mean term barely moves.

The worked example in a 2D feature space. Real data: μ₁ = (0, 0), Σ₁ = [[1,0],[0,1]]. Model: μ₂ = (1, 1), Σ₂ = [[4,0],[0,1]].

Step 1 — the mean term.

Step 2 — the product inside the square root. Since Σ₁ = I:

It is diagonal, so its matrix square root is the elementwise sqrt of the diagonal:

Step 3 — assemble the trace term.

Step 4 — add the terms.

Interpretation: 2 of the 3 units come from the clouds being centered a diagonal step apart; 1 unit comes from the model over-spreading the x-direction (std 2 vs 1) while matching y exactly (its y-contribution: 1 + 1 − 2×1 = 0).

Step 5 — the correlated case, which is no longer hand-friendly — this is why sqrtm exists. With Σ₂ = [[2, 0.6],[0.6, 1]], scipy returns sqrtm with off-diagonal 0.2544 and:

Step 6 — the Inception Score mini example. IS = exp(E[KL(p(y|x) ‖ p(y))]): samples should be individually recognizable (sharp conditionals p(y|x)) and collectively varied (uniform marginal p(y)). Two images, two classes: p(y|x₁) = (0.9, 0.1), p(y|x₂) = (0.1, 0.9). The marginal p(y) = (0.5, 0.5).

Image 2 is identical by symmetry, so:

Sharp and varied, so IS exceeds 1. Had both images produced (0.9, 0.1), the marginal would equal each conditional, KL = 0, IS = 1 — the varied-ness is gone.

python
# From scratch: 2x2 FID via eigendecomposition for the sqrt, printing the
# eigenvalues and reassembled root, plus the Inception Score mini example.
import numpy as np
def fid_2x2(mu1, S1, mu2, S2):
    mean_term = float(np.sum((mu1 - mu2)**2))
    prod = S1 @ S2
    vals, vecs = np.linalg.eigh(prod)
    root = vecs @ np.diag(np.sqrt(np.clip(vals, 0, None))) @ vecs.T
    print(f"  eig(S1 S2)={vals.round(3)}  sqrt={np.sqrt(vals).round(3)}")
    trace_term = float(np.trace(S1 + S2 - 2*root))
    return mean_term, trace_term, mean_term + trace_term

print(fid_2x2(np.zeros(2), np.eye(2),
             np.ones(2),  np.diag([4., 1.])))   # (2.0, 1.0, 3.0)

pyx = np.array([[0.9,0.1],[0.1,0.9]]); py = pyx.mean(0)
kl = (pyx * np.log(pyx/py)).sum(1)
print(round(float(kl[0]),4), round(float(np.exp(kl.mean())),4))  # 0.3681 1.4449

python
# Library: the real thing uses scipy.linalg.sqrtm and takes the real part.
import numpy as np
from scipy.linalg import sqrtm
mu1, mu2 = np.zeros(2), np.ones(2)
S1 = np.eye(2); S2 = np.array([[2, 0.6], [0.6, 1.]])
fid = np.sum((mu1-mu2)**2) + np.trace(S1 + S2 - 2*np.real(sqrtm(S1 @ S2)))
print(round(float(fid), 4))   # 2.2835  (correlated case)

FID's fine print — the things papers get burned by. It is biased upward at small n (fewer samples inflate FID; compare only at matched counts, conventionally 50k vs 50k). It depends entirely on the feature network (Inception features encode ImageNet-classification semantics — hence CLIP-feature FID variants in recent text-to-image work; see contrastive-clip). And a single-number FID hides where the mismatch is.

Inception Score's obsolescence, and two more instruments. IS has a fatal flaw: it never looks at real data — a model producing one perfect image per class maxes IS while ignoring the data distribution. The field moved to FID for exactly this reason. CLIPScore handles text-conditioned generation: cosine similarity between CLIP embeddings of the prompt and the generated image, measuring prompt alignment — orthogonal to FID's distribution match (a model can ace FID while ignoring prompts). And precision/recall for generative models (Kynkäänniemi et al.) factorizes FID's single number into the two Chapter-1 axes: precision = fraction of generated samples inside the real manifold (fidelity); recall = fraction of real samples covered (diversity). Mode collapse shows up as high precision / low recall; hallucination as the reverse.

FID scored whole sample distributions. But vision-language outputs — captions, VQA answers — are graded against many acceptable answers, one image at a time. Ask ten people what's in a photo and you get ten different, mostly-correct sentences. Grading a VLM means grading against that fuzzy cloud. Next chapter.

Chapter 6: Evaluating VLMs — VQA, Captions, Hallucination

Ask ten people what's in a photo and you'll get ten different — mostly correct — sentences. Grading a vision-language model means grading against that fuzzy cloud of acceptable answers, not one gold string.

Make the many-right-answers problem concrete. For one image, "a man riding a horse on a beach," "someone on horseback by the sea," and "a rider gallops along the shore" are all correct captions sharing almost no exact words. Exact-match dies immediately. Every VLM metric is a scheme for partial credit against multiple references.

VQA accuracy — the consensus protocol. The VQA v2 dataset collects 10 human answers per question. A model answer scores min(#matching humans / 3, 1): agree with 3+ humans for full credit, partial credit below. (The official metric averages this over all 10-choose-9 annotator subsets; we teach the simple form.) The design rationale: answer normalization plus the /3 threshold encode "consensus defines correctness" when no single string is the answer.

The yes-bias trap. Yes/no questions are ~40% of VQA, and models learn the prior ("is there a …" is usually "yes" in crowdsourced questions). A model can score deceptively well by exploiting answer priors without looking at the image — which motivated VQA v2's balanced complementary pairs and leads directly into POPE below.

BLEU, built up. Modified n-gram precision = clipped candidate n-gram counts / total candidate n-grams (clipping stops "the the the" from farming credit); geometric mean across n-gram orders; a brevity penalty punishes short outputs, since precision alone rewards saying almost nothing. Play with a live caption grader before the arithmetic.

The worked example, three parts. Part 1 — BLEU-2 for candidate "the cat sat on the mat" vs reference "the cat is on the mat."

Step 1 — unigram precision with clipping. Candidate tokens: the, cat, sat, on, the, mat (6 tokens; "the" appears twice). Clip each match against the reference's counts (the×2, cat, is, on, mat):

Step 2 — bigram precision. Candidate bigrams: (the cat), (cat sat), (sat on), (on the), (the mat) = 5. Reference bigrams: (the cat), (cat is), (is on), (on the), (the mat). Matches: (the cat), (on the), (the mat) = 3.

Step 3 — brevity penalty. Candidate length 6 = reference length 6, so BP = 1.

Step 4 — combine (geometric mean).

Part 2 — VQA accuracy. The model answers "yellow," which 2 of 10 annotators gave:

Part 3 — POPE hallucination probing. POPE turns "does it hallucinate objects?" into balanced binary probes: "Is there a dog?" over object-present sets (from ground-truth annotations) and object-absent sets (negatives sampled to catch prior-exploitation). On 50 present and 50 absent probes, the model says yes 45 and 15 times:

On a 50/50-balanced probe set, an image-grounded model's yes-rate should be near 0.50. Here it is 0.60 — the model says yes 10 points too often; recall is being purchased with hallucinations, exactly the pattern POPE's balanced design exposes.

Why BLEU fails captions, and the two fixes. BLEU treats "a man rides a horse" vs "a person on horseback" as near-zero overlap (synonyms invisible), and rewards safe generic captions that share stopwords with every reference. CIDEr fixes credit assignment by weighting each n-gram with TF-IDF across the reference corpus — distinctive consensus phrases ("riding a horse") count heavily, boilerplate ("there is a") counts near zero; the score is average cosine similarity of TF-IDF n-gram vectors (it needs corpus statistics, so scores are corpus-relative). SPICE fixes surface-form dependence by parsing candidate and references into scene graphs and computing F1 over object/attribute/relation tuples — "a young girl" matches "a small child" at the tuple level — but inherits every parser error.

python
# From scratch: BLEU-2 with clipping, VQA soft accuracy, POPE scores.
from collections import Counter
import math

def ngrams(toks, n): return Counter(zip(*[toks[i:] for i in range(n)]))
def bleu2(cand, ref):
    c, r = cand.split(), ref.split()
    p = []
    for n in (1, 2):
        cn, rn = ngrams(c, n), ngrams(r, n)
        clipped = sum(min(v, rn[g]) for g, v in cn.items())  # clip = Counter intersect
        p.append(clipped / sum(cn.values()))
        print(f"  {n}-gram: {clipped}/{sum(cn.values())} = {p[-1]:.4f}")
    bp = 1.0 if len(c) >= len(r) else math.exp(1 - len(r)/len(c))
    return bp * math.exp(sum(math.log(x) for x in p)/len(p))

print(round(bleu2("the cat sat on the mat", "the cat is on the mat"), 4))  # 0.7071
print(round(min(2/3, 1), 4))                                          # 0.6667 VQA
p, r = 45/60, 45/50
print(p, r, round(2*p*r/(p+r), 4), (45+15)/100)                    # 0.75 0.9 0.8182 0.6

python
# Library: the full pipeline via sacrebleu / pycocoevalcap (BLEU/CIDEr/SPICE).
from pycocoevalcap.cider.cider import Cider
from pycocoevalcap.spice.spice import Spice
# scorer.compute_score({id: [refs]}, {id: [candidate]}) -> corpus-level score

Every metric so far — judges, VQA consensus, caption scores — is ultimately calibrated against humans. So what calibrates the humans? Two raters agreeing 70% of the time sounds decent, until you compute how much of that agreement is pure luck. Next chapter.

Chapter 7: Human Eval Done Right — Agreement, Protocol, Cost

Humans are the gold standard that judges and metrics are calibrated against — so what calibrates the humans? Two raters agreeing 70% of the time sounds decent, until you compute how much of that agreement is luck.

Reframe humans as instruments. Everything in Chapters 3–4 treated the vote as data; this chapter asks how good the votes are. The core insight: raw percent agreement is inflated by chance agreement. Two raters who both label "acceptable" 90% of the time agree 82% of the time by coin-flipping alone. Any honest agreement statistic must subtract the luck.

Derive Cohen's kappa from that requirement. Let p_o be the observed agreement (fraction of items where the raters match) and p_e the expected-by-chance agreement (the probability they'd match if each rated independently at their own base rates — a sum over categories of the product of the two marginals). Then:

In words: "the fraction of the achievable beyond-chance agreement that was actually achieved." If p_e already gives you agreement, only what you earn above it counts, scaled by how much room there was.

Play with the agreement matrix before the arithmetic. Drag the four cells; watch p_o (the diagonal), the ghost "chance matrix" p_e, and κ on the gauge. The presets include the worked example and the notorious kappa paradox.

The worked example: two raters label 100 model responses acceptable/unacceptable. Both say acceptable: 40. Both say unacceptable: 30. R1 acceptable / R2 unacceptable: 20. R1 unacceptable / R2 acceptable: 10.

Step 1 — observed agreement is the diagonal.

Step 2 — marginals. R1 says acceptable 40 + 20 = 60 times (0.60); R2 says acceptable 40 + 10 = 50 times (0.50). So R1 unacceptable 0.40, R2 unacceptable 0.50.

Step 3 — chance agreement.

Half their agreement was free.

Step 4 — kappa.

Of the 0.50 of agreement available beyond luck, the raters achieved 0.20 — 40%. "Moderate" on the Landis-Koch convention, and typical for open-ended quality judgments.

Step 5 — the kappa paradox. Keep p_o = 0.70 but make marginals extreme (both raters say acceptable 90%):

The same raw agreement is now worse than chance. Marginals matter as much as matches.

Calibrate the scale honestly. The Landis-Koch adjectives (0.2 slight … 0.8 almost perfect) are conventions, not laws — report the number. Real generative-eval kappas are humbling: open-ended quality judgments often land at 0.3–0.5 between trained raters. And that number is the ceiling for any automated metric: a judge cannot agree with humans more than humans agree with each other. So measure human-human kappa first — it is the denominator of every "our judge matches humans" claim.

Kappa's edge cases and upgrades. High raw agreement with skewed marginals gives shockingly low kappa (the paradox). Cohen's kappa is for two raters on nominal labels. The upgrades, with triggers: Fleiss' kappa (3+ raters), weighted kappa (ordinal scales where off-by-one disagreement is milder), Krippendorff's alpha (any number of raters, missing data, any scale — the Swiss-army option). Full derivations belong to eval-statistics.

Protocol — three pillars. (1) Blinding: raters must not know which system produced an output; strip model names, formatting tells, watermarks, and even indirect tells (characteristic phrasing, emoji habits). Position randomization per item is blinding's twin for pairwise tasks. (2) Instructions and rubrics: anchor every scale point with a concrete example ("3 = factually correct but omits a key caveat"); pilot the rubric on 20 items, measure kappa, and revise the rubric before scaling — low pilot agreement is a rubric bug, not a rater bug. (3) Attention checks and honeypots: seed known-answer items, track per-rater honeypot accuracy and per-rater kappa-vs-pool, drop or retrain raters below threshold. On crowdsourcing platforms expect to filter 10–30% of raters; cap session length and shuffle item order per rater to decorrelate fatigue from difficulty.

The cost-quality frontier, with real arithmetic: raters × items × wage. 1000 pairs × 3 raters × $0.50 = $1,500 and days of latency, versus ~$4 of judge calls for the same volume (Chapter 9 does this arithmetic) — the entire economic case for Chapter 4's calibrated judges. The design pattern that gets both: small, protocol-rigorous human sets to (a) measure the human-human ceiling and (b) calibrate/audit the judge; the judge then scales to millions. Agreement statistics are the exchange rate between the two layers.

python
# From scratch: Cohen's kappa with the confusion table, marginals, pe, kappa.
def cohens_kappa(both_acc, r1acc_r2un, r1un_r2acc, both_un, n=100):
    po = (both_acc + both_un) / n
    r1_acc = (both_acc + r1acc_r2un) / n            # rater-1 marginal
    r2_acc = (both_acc + r1un_r2acc) / n            # rater-2 marginal
    pe = r1_acc*r2_acc + (1-r1_acc)*(1-r2_acc)      # chance agreement
    print(f"  po={po:.2f}  pe={pe:.2f}")
    return (po - pe) / (1 - pe)

print(round(cohens_kappa(40, 20, 10, 30), 4))   # 0.4
print(round(cohens_kappa(63, 27, 3, 7), 4))     # -0.6667  (paradox: po=0.7, skewed)

python
# Library: sklearn agrees; krippendorff.alpha for the general case.
from sklearn.metrics import cohen_kappa_score
r1 = [1]*40 + [1]*20 + [0]*10 + [0]*30
r2 = [1]*40 + [0]*20 + [1]*10 + [0]*30
print(round(cohen_kappa_score(r1, r2), 4))   # 0.4

You have now derived every instrument — verifiers, preference aggregation, judge calibration, distribution metrics, and the human ceiling. Time to run the whole pipeline against a rigged judge and rescue the leaderboard yourself. The showcase.

Chapter 8: Showcase — The Judge-Bias Lab

You have derived every instrument. Now run the whole pipeline against a rigged judge and rescue the leaderboard yourself. Unlike the real world, this laboratory knows the ground truth — and that is what makes the lesson visible.

The laboratory contract. Model A truly beats model B on 55% of prompts (a planted quality gap). The judge is simulated with three tunable corruptions from Chapter 4: position bias (+b to whichever answer is first), verbosity bias (+γ per 100 extra tokens, and model B writes longer), and self-preference (+s when judging its own family — B shares the judge's family). Your job: measure the truth through the corrupted instrument.

The showcase runs the full pipeline live. Set the red bias dials, flip the green repair switches, and move the battle-count slider; the leaderboard re-simulates every change, with bootstrap CI whiskers and a truth marker. The status banner grades your pipeline: CONFIDENTLY WRONG, NOISY, or TRUSTWORTHY.

Walk the six acts. The worked numbers: true P(A beats B) = 0.55, judge position bias b = 0.10, layout A-always-second.

Act 1 — naive pipeline, confident lie. A's answers are always placed second (alphabetical-by-run-id, the classic accidental systematic layout). A is never first, so:

B tops the leaderboard, and with enough battles the CI excludes 0.5 — statistically significant, systematically wrong. The flip threshold: the ranking inverts whenever b > 0.05 (the true margin). A bias as small as 6pp beats a 5pp true quality gap.

Act 2 — the flip-rate audit. Run the Chapter 4 swap experiment inside the lab; the flip rate and position effect surface immediately. The instrument fails acceptance; every leaderboard it produced is quarantined. The audit costs 2× judging on a few hundred pairs — run it before the 50,000-battle campaign, not after.

Act 3 — repair 1, randomization. Per-battle coin-flip ordering makes the expected position term cancel:

The measured rate re-centers on truth as the toggle flips; the position dial now moves nothing. Design beats correction — no estimate of b was ever needed.

Act 4 — repair 2, length. Verbosity bias reintroduces a tilt through B's longer answers. Two remedies: (i) length-matched subsetting (only battles with |length diff| < threshold — unbiased but discards data), or (ii) length-controlled adjustment (fit judged preference against length difference, read the quality term at zero gap — the LC-AlpacaEval idea). Either holds the leaderboard steady as the verbosity dial sweeps.

Act 5 — repair 3, jury. With self-preference s active, a single same-family judge inflates B. Swap in a 3-judge jury (different families, majority vote) or exclude the same-family judge; the s dial's leverage collapses. Multi-judge juries and family-exclusion rules are now standard practice in serious eval reports.

Act 6 — now, and only now, sample size. With bias handled, how many battles until the CI separates 0.55 from 0.50? The 95% two-sided margin at worst-case variance (p = 0.5):

The sobering leaderboard version: a 10-Elo-point gap implies win probability 1/(1 + 10^−10/400) = 1/(1 + 0.9441) = 0.5144, so the margin to resolve is 0.0144:

That is 12× the ±5pp case for a gap 3.5× smaller — because n scales as 1/margin². This is why arena leaderboards need hundreds of thousands of votes to order a dense model field.

The order of operations, numerically. At n = 5,000 with the rigged layout, the CI around the biased 0.45 has half-width:

The interval excludes 0.5 decisively, "proving" the wrong winner. More data made the lie stronger. Bias first, then n.

python
# The full lab: simulate one battle, run a campaign, bootstrap the CI.
import numpy as np
rng = np.random.default_rng(0)

def battle(true_p, b, g, s, randomize, length_ctrl, jury):
    a_first = rng.random() < 0.5 if randomize else False  # A second by default
    p = true_p + (b if a_first else -b)          # position term
    if not length_ctrl: p -= g                    # B is longer -> tilts against A
    if not jury:        p -= s                    # same-family judge favors B
    return rng.random() < min(max(p, 0), 1)

def campaign(n, **kw):
    wins = np.array([battle(0.55, **kw) for _ in range(n)])
    boot = [rng.choice(wins, n).mean() for _ in range(1000)]
    return wins.mean(), np.percentile(boot, [2.5, 97.5])

rate, ci = campaign(5000, b=0.10, g=0, s=0,
                    randomize=False, length_ctrl=True, jury=True)
print(round(rate, 3), ci.round(3))   # ~0.45 [0.436 0.464] -- confidently WRONG

python
# The sample-size arithmetic every eval-design answer needs.
print(round(1.96**2 * 0.25 / 0.05**2, 1))          # 384.2  (+/-5pp)
E = 1 / (1 + 10**(-10/400))
print(round(E, 4), round(1.96**2 * 0.25/(E - 0.5)**2))   # 0.5144 4640

The lab just used Chapter 0's flip demo, Chapter 3's BT/win-rate machinery, Chapter 4's calibration experiments, and Chapter 7's protocol thinking against one adversary. The same drill generalizes — FID pipelines get "rigged instrument" failures too (mismatched preprocessing = a rigged feature extractor). Next, we compress all of it into interview-ready form.

Chapter 9: Interview Arsenal — Generative Eval Under Fire

Everything you derived is now a target. Staff interviews probe exactly the places where naive answers and correct answers diverge — plug-in pass@k, raw win rates, tight CIs on biased pipelines.

The interview landscape clusters into four archetypes: (1) derive/compute the estimator (pass@k, kappa, FID term-by-term), (2) this number looks great, why might it be a lie? (the bias catalog), (3) design the eval for product X (protocol + cost), (4) metrics disagree, arbitrate (metric semantics). Each maps to the chapter that armed you.

The two-minute whiteboard drills and their canonical numbers, rehearsed until muscle memory — interviewers watch for hesitation at p_e and at C(n−c, k):

The bias catalog as a recitable checklist, each with its one-line detection experiment: position, verbosity, self-preference (judges → swap/pad/blind); contamination/memorization (likelihood → n-gram overlap); small-n bias (FID and plug-in pass@k → matched-n); opponent-strength confounding (raw win rates → BT fit); chance agreement (raw rater agreement → kappa); yes-prior exploitation (VQA/POPE → balanced probes, yes-rate).

The ceiling argument as a reusable move: any claim "our judge/metric achieves X% agreement with humans" is meaningless without the human-human number; the ratio (e.g., kappa 0.42/0.45 = 93%) is the honest statistic. Interviewers reward candidates who ask for the denominator unprompted.

The full Q&A bank — each with the follow-up an interviewer would actually ask next. Open each for the model answer.

The cost-architecture drill every eval-design answer needs. Compare evaluating 1,000 response pairs three ways.

The resulting three-layer architecture: (i) free automated metrics on every commit as regression alarms, (ii) the calibrated judge on 10k–100k pairs for leaderboards and launch decisions, (iii) the $1,500-grade human protocol quarterly and on disagreement hot-spots to re-measure the ceiling and re-audit the judge. Both ratios (375×, 93.3%) belong in the interview answer.

Three debate prompts — know both defensible sides.

You can now derive every estimator under time pressure. The last step is knowing which neighboring lesson to open when a real eval problem exceeds this one's borders. Connections.

Chapter 10: Connections — Where This Math Plugs In

You now own the math. The last step is knowing which neighboring lesson to open when a real eval problem exceeds this one's borders.

Draw the family map from your position. This lesson sits in the middle of the Evaluation & Analytics family: metric-design (prereq) decided what to measure; this lesson derived how to estimate it for generative outputs; eval-statistics generalizes the CI/bootstrap/power machinery used informally here; eval-plots renders these numbers honestly; experiment-design takes preference data online (A/B); regression-testing-ml turns any of these metrics into CI gates; metrics-ladder rolls them up for execs.

The decision procedure to close the lesson — drag a scenario onto the tree and follow the branches. This is the mental model for any new generative-eval problem.

The cross-link table — which sibling owns each neighboring concern.

Where the field is moving, so the lesson ages well: style- and length-controlled preference modeling (2024-25 arena updates), multi-judge juries replacing single judges, verifier-based rewards blurring eval into training (RLVR), and distribution metrics migrating from Inception to CLIP/DINO features — all are refinements of the estimators here, not replacements. The audit mindset transfers unchanged.

One judge-adequacy recap that ties Ch4 + Ch7 + Ch9 into a reusable rule. Judge-vs-human kappa 0.42, human-human kappa 0.45, post-randomization flip rate 4%. May the judge replace humans?

Generalize: license = (ceiling ratio ≥ 0.9) AND (all Ch4 bias audits within bars) AND (a standing human re-audit budget). Any "can we automate this eval?" question in any modality reduces to instantiating these three checks.

The cheat sheet — every key formula, what each symbol means, and when to use it.

The send-off: you can now derive pass@k under time pressure, fit a leaderboard from votes, price a human study, calibrate a judge, and — most importantly — refuse to trust any generative-eval number until you know its estimator, its bias exposure, and its error bar. That refusal is the whole discipline.