Mode Collapse

Chapter 1: Infinity-Chat: Building a Benchmark Diversity Can Actually Be Measured On

Chapter 0 leaned on a number — 0.800 versus 0.837 — without explaining where the underlying dataset came from, or why building it was itself a nontrivial research contribution. Open it up.

Building a trustworthy benchmark for something as slippery as “how open-ended is this question” is its own research problem, separate from anything about model behavior. Get the benchmark wrong — too synthetic, too narrow, not validated against real human judgment — and every downstream number in this session, no matter how carefully computed, would be measuring an artifact of the dataset rather than a real property of language models. This chapter is the paper's answer to that problem, and it is worth taking as seriously as any of the numeric results that depend on it.

Why the old benchmarks couldn't answer this question

Before this paper, most attempts to measure LM creativity used narrow, synthetic proxies: ask a model to generate a random number, name a random word, or write in a fixed persona, then measure how repetitive the outputs are. These tasks are easy to score, but they are not what people actually ask chatbots. A benchmark built entirely from “pick a random number between 1 and 100” tells you almost nothing about whether a model runs out of ideas when asked to brainstorm a birthday gift, write a short story, or explain a concept in an unusual way — the actual bulk of real open-ended chatbot traffic.

The paper's answer is Infinity-Chat: a dataset built not from synthetic prompts invented by researchers, but mined from WildChat, a large public corpus of real conversations people had with GPT-4 in the wild. Using real traffic instead of invented prompts matters because it guarantees the benchmark reflects what people actually ask, not what a research team assumes they ask.

Specifically, the raw material is allenai/WildChat-1M — a public HuggingFace release of one million real ChatGPT conversations, donated by users who opted in to having their chats logged for research. That scale matters for a filtering pipeline: you can afford to be strict at every pass and still end up with tens of thousands of usable queries, because you started with a million.

The filtering funnel, with the arithmetic shown

Turning raw chat logs into a clean open-ended benchmark takes several filtering passes, each one narrowing the pool. Start with the raw candidate pool and work through it exactly as the paper did.

Pass 1 — basic quality filters. From the full WildChat-1M release, keep only queries that are (1) written in English, (2) flagged non-toxic and non-harmful by WildChat's own built-in labels, (3) directed at GPT-4 specifically, and (4) of moderate length — between 15 and 200 characters, long enough to be a real question, short enough to stay a single clear ask rather than a multi-part essay prompt. This yields:

Pass 2 — semantic classification. Every one of those 37,426 candidates is then run through gpt-4o-2024-11-20, which labels each along three axes: does it seek meaningful information (versus being gibberish); is it a greeting or a question about the model itself (“are you an AI?”); and does it admit a single specific answer or many valid ones. Queries that are unclear get lightly rewritten for clarity in the same pass, never dropped just for being awkwardly phrased.

Look closely at those three axes, because the second one is doing more work than it first appears. Meaningful information asks whether the query poses a real question or seeks substantive information at all — this is what catches gibberish. Greeting or model inquiry catches queries like “are you an AI?” or “hello” that are not really asking for content generation, and would otherwise pollute an “open-ended” bucket with conversational noise that has nothing to do with creative or analytical diversity. Response type is the axis that actually decides open-ended versus closed-ended: does this query admit a single specific correct answer, or many valid, diverse responses? A query can pass the first two filters and still fail the third — “what is the capital of France?” seeks meaningful information and is not a greeting, but it has exactly one correct answer, so it lands in the closed-ended pile.

Seeing real examples from each pile makes the boundary concrete. The paper's appendix lists actual queries mined from WildChat that landed on each side of the classifier:

Read the left column against the right and the pattern the classifier is actually detecting becomes visible: the open-ended queries ask for a kind of output (tips, a rave, an explanation, a story) with no fixed correct instance, while the closed-ended queries ask a question that resolves to a specific, checkable fact. Notably, even a seemingly narrow request — “3 short tips for self-development” — counts as open-ended, because there is no single correct set of three tips, only many equally valid ones. The boundary is about whether the space of correct answers has one member or many, not about how constrained or specific the prompt's phrasing sounds.

Notice those two numbers do not sum back to 37,426. The remainder — greetings, model-inquiries, and genuinely non-meaningful text — gets discarded entirely:

So of the original candidate pool: 69.7% became open-ended queries (26,070 ÷ 37,426), 23.6% became closed-ended queries with a single expected answer (8,817 ÷ 37,426), and 6.8% were noise. Infinity-Chat is the 26,070-query open-ended slice — the dataset this entire session is built on, rounded to “26K” in the paper's own framing.

Checking the filter's own honesty: human validation

A classifier deciding what counts as “open-ended” is only useful if it agrees with actual humans. The authors sampled 100 queries from Infinity-Chat, recruited 86 Prolific participants (English-fluent, approval rate above 99%, at least 500 prior submissions, some post-secondary education — a genuinely screened pool, not an anonymous convenience sample), and assigned three annotators per query to answer two questions: is this open-ended at all, and if so, roughly how many different reasonable answers does it admit.

That last row is the exact source of Chapter 0's “high-open-endedness” group: for roughly one in three queries in this dataset, real humans independently judged that more than twenty genuinely different good answers exist. This is not a benchmark built from cherry-picked ambiguous trivia — it is, by the humans' own accounting, a dataset dominated by questions with wide-open answer spaces.

Infinity-Chat100: the working subset every later chapter runs on

26,070 queries is far too many to run every experiment against — generating 50 responses from 25 models for every single one of them would mean 25 × 50 × 26,070 = 32.6 million generations for Chapter 2's intra-model study alone. So the paper curates a smaller, representative, human-verified working set: Infinity-Chat100, exactly 100 queries drawn from Infinity-Chat and confirmed open-ended by the same human-validation process described above. Every core measurement from here forward — the intra-model histograms of Chapter 2, the min-p comparison of Chapter 3, the inter-model clustering of Chapter 4, the absolute-rating annotation of Chapter 7 — runs on some slice built from this 100-query set, not the full 26,070. Keep that scoping in mind: every headline statistic in this session is a measurement on a carefully chosen, human-checked 100-query sample of the larger 26,070-query benchmark, not a claim about all 26,070 queries individually.

What kinds of questions people actually ask

Classifying 26,070 queries by hand is not feasible, so the same gpt-4o-2024-11-20 classifier assigns each query to one or more categories from a taxonomy the authors built by hand from an initial sample of about 100 queries, then iteratively refined into a hierarchy: 6 top-level categories comprising 17 fine-grained subcategories. Because a single query can span multiple categories, the percentages below do not sum to 100%.

Six top-level buckets, seventeen leaves — count them: 1 (Creative) + 5 (Open-Endedness) + 6 (Information-Seeking) + 2 (Alternative Styles) + 1 (Brainstorm & Ideation) + 2 (Alternative Perspectives) = 17. The classifier also surfaced 314 entirely new categories beyond the initial 17, most prominently clustered around themes like “cultural,” “ethical,” “historical,” and “humor” — a sign the taxonomy captures the bulk of real usage without claiming to be exhaustive.

Notice which category should worry you most: Brainstorm & Ideation sits at only 15.2%, smaller than Creative Content Generation or Writing Genres — but it is the category where a user is explicitly asking the model to generate new ideas, not polish existing ones. If mode collapse hits this category hardest, the harm is not stylistic sameness in prose; it is a measurable narrowing of what ideas a large fraction of users are even exposed to when they ask an AI to help them think of something new.

The model roster and the infrastructure behind every later number

Every diversity measurement in this session depends on actually running a lot of different language models under a controlled, identical protocol, so it is worth seeing the roster once, in full, rather than meeting individual model names piecemeal in later chapters. The paper curates 70+ models in total (25 of them detailed in the main paper's figures, selected as the strongest or largest model from each family to keep the main text readable), spanning open-source families — Llama-3.1, Llama-3.2, and Llama-3.3 at multiple parameter counts; Gemma-1.1 and Gemma-2; five generations of Qwen (1.5, 2, 2.5, 3) spanning 0.5B to 110B parameters; DeepSeek-V3; the Mistral, Ministral, and Mixtral lines; Phi-3 and Phi-4; Cohere's Aya-Expanse and Command-R lines; and Allen Institute's own OLMo-2 and Tulu-3 — alongside closed-source flagships: multiple GPT-4o snapshots plus GPT-4-Turbo, multiple Claude-3 and Claude-3.5 models, Gemini-1.5 and Gemini-2.0, and Alibaba's hosted qwen-max/qwen-plus/qwen-turbo API models.

The generation infrastructure itself is heterogeneous, and deliberately so: open-source HuggingFace checkpoints run on the authors' own NVIDIA A100 or H100 GPUs; closed-source models (OpenAI, Anthropic, Gemini, Qwen's hosted API) go through their respective vendor APIs; and a handful of the largest open-weight models — DeepSeek-V3 chief among them — exceed what the authors' own GPUs can serve, so those go through TogetherAI's hosted inference instead. Regardless of which of the three paths a given model's responses came from, every model in the study is held to the exact same decoding configuration: top-p = 0.9, temperature = 1.0, maximum generation length 2,048 tokens, 50 independent samples per query. That uniformity is what makes a claim like “79% of (model, query) pairs exceed 0.8 similarity” a fair comparison across models built by different companies on different hardware — the only thing that varies is the model itself, never the sampling procedure that generated its outputs.

In code: the funnel, computed

python
candidates = 37_426
open_ended = 26_070
closed_ended = 8_817
discarded = candidates - (open_ended + closed_ended)

print(round(100 * open_ended / candidates, 1))    # 69.7 -- % kept as open-ended (Infinity-Chat)
print(round(100 * closed_ended / candidates, 1))  # 23.6 -- % kept as closed-ended (single-answer)
print(discarded, round(100 * discarded / candidates, 1))  # 2539  6.8 -- greetings/gibberish/model-inquiries, dropped

Chapter 2: Two Flavors of Collapse

With a validated 26,070-query benchmark in hand, the paper narrows to a curated, human-verified subset of 100 representative queries — called Infinity-Chat100 — and uses it to run the core experiment of the whole study. It measures two conceptually distinct failure modes, and keeping them separate in your head matters, because the fixes for each turn out to be different.

Defining the two flavors

Intra-model repetition is one model, asked the same question many times, giving back nearly the same answer every time — the “intra” prefix means within a single model. Inter-model homogeneity is different models — different companies, different architectures, different training data — converging on the same answer to the same question, the “inter” prefix meaning between models. A model could in principle be diverse against itself but still land on the same handful of ideas everyone else lands on; the paper measures both, separately, and it turns out both are real, and the second is arguably worse, because it means ensembling models does not rescue you the way you'd hope.

Keep both definitions precise as you read the next three chapters, because it is easy to blur them together under the single word “collapse.” This chapter measures only the first kind. Chapter 4 measures the second. They use the same underlying tool (embed, then compare with cosine similarity) but answer genuinely different questions, and a fix for one is not automatically a fix for the other — which is exactly why Chapter 3's decoding-time intervention, tested against intra-model repetition, does not settle anything about inter-model homogeneity at all.

The measurement protocol, in full

For every one of the 100 Infinity-Chat100 queries, each of 25 flagship models (spanning GPT-4o variants, Claude, Gemini, DeepSeek-V3, Qwen, Llama, and more — the strongest model from each of several major families, with 70+ models tested in total across the paper's full appendix) generates 50 independent responses, using standard nucleus sampling: top-p = 0.9, temperature = 1.0, up to 2,048 tokens. That is 25 × 50 = 1,250 responses per query, 100 queries deep.

Multiply that out across the full protocol and the scale of the underlying experiment is worth stating plainly: 1,250 responses per query × 100 queries = 125,000 total generations, feeding a downstream analysis pipeline of pairwise cosine-similarity computations, PCA projections (Chapter 4), and top-N clustering (also Chapter 4) — all of it built from the same 125,000 raw text generations. Every headline percentage in this session traces back to that one shared pool of responses, computed differently depending on which question is being asked of it.

Every response gets converted into a sentence embedding — a single vector that captures the response's meaning — using OpenAI's text-embedding-3-small model. This is the direct bridge to two earlier lessons in this course: if you have not internalized what an embedding vector is and how cosine similarity turns two of them into a single number between −1 and 1, go back to vector-embeddings.html and similarity-metrics.html before continuing — everything past this point is built on that one operation, applied at massive scale.

python
import numpy as np

def cosine_similarity(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

# for one query: 50 responses from one model -> 50 embedding vectors
embeddings = get_embeddings(responses)         # shape: (50, 1536) for text-embedding-3-small
pairs = [(i, j) for i in range(50) for j in range(i+1, 50)]
sims = [cosine_similarity(embeddings[i], embeddings[j]) for i, j in pairs]
avg_intra_model_similarity = sum(sims) / len(sims)   # one number, per (model, query) pair

Repeat that computation for every (model, query) pair and you get a distribution of average intra-model similarities — one per model per query, 25 × 100 = 2,500 numbers in total — which is exactly what gets bucketed into the histogram below.

The random baseline, which makes the result mean something

A skeptical reader's first objection should be: maybe 0.8 cosine similarity is just what “normal, unrelated English text” looks like in this embedding space, and the whole result is a measurement artifact. The paper closes this exact loophole with a control: take randomly paired responses from the entire global response pool (different queries, different models, no relationship at all) and compute the same similarity.

This is a textbook control condition, the same experimental-design principle that underlies every clean scientific comparison: to know whether a measured effect is real, you need to know what the measurement would show if the effect you are testing for were entirely absent. Here, the effect under test is “same model, same prompt produces suspiciously similar responses.” The control asks what cosine similarity looks like when that specific relationship (same model, same prompt) is deliberately broken — by pairing responses that share neither. Without this step, a critic could reasonably claim the 0.8 threshold was arbitrary, chosen after the fact to make the result look dramatic. With it, the threshold is anchored to something independently meaningful: a number four to eight times higher than what unrelated text produces in the same embedding space, under the same embedding model, computed by the exact same code.

The headline numbers, with an honest caveat about which models they cover

The paper's main-text figure, built from the 25 flagship models, reports: in 79% of (model, query) cases, average pairwise response similarity exceeds 0.8. The appendix additionally reports the full per-model breakdown across every model tested (70+, including many smaller open-source checkpoints down to 0.5B parameters) — and averaged across that much larger and more varied pool, the top two similarity buckets (0.8–0.9 and 0.9–1.0) sum to about 70%. Both numbers are real; they are simply different denominators — the flagship-only 79% figure versus the full-corpus average that includes weaker models whose sampling tends to be noisier. Either way you slice it, a strong majority of the time, one model asked the same open-ended question fifty times gives back fifty near-duplicates.

Two specific flagship models from the appendix's full table make this concrete. For gpt-4o-2024-11-20, across its 100 (query, 50-response) trials: 51% land in the 0.9–1.0 bucket and 36% in 0.8–0.9, for 51 + 36 = 87% of trials exceeding 0.8 similarity. For claude-3-5-sonnet-20241022: 61% in 0.9–1.0, 22% in 0.8–0.9, for 61 + 22 = 83% exceeding 0.8. Both flagship models sit well above the broader corpus average — the strongest, most widely deployed models are not the exception to this problem. If anything, they show it more cleanly.

Does scale change the picture? A worked comparison across the appendix table

The paper's appendix (Table 6) reports this same “percent of trials exceeding 0.8 similarity” statistic for every one of the 70+ models tested, not just the two flagships quoted above. Reading down that table by parameter count turns up a pattern worth building yourself, arithmetic and all, even though the paper's main text does not frame it this way explicitly — treat what follows as this lesson's own reading of the appendix numbers, not a causal claim the paper itself makes.

Read down the right-hand column and a rough trend appears: the smallest, weakest model in this slice (Llama-3.2-1B-Instruct, at 39%) sits far below every model 70 billion parameters and up, which cluster tightly between 81% and 87%. That is not proof that scale causes tighter mode collapse — the paper's own main text offers a more cautious reading, noting elsewhere that weaker models' sampling “tends to be noisier,” which would also push their similarity scores down for reasons that have nothing to do with creativity (an under-trained 1B model may simply produce more incoherent, off-topic completions, which embed far apart from each other by accident rather than by genuine idea-diversity). Still, the direction is consistent across nine data points spanning three orders of magnitude in parameter count, and it is worth carrying forward as an open question: does becoming a better, more capable, more heavily-aligned model also make a model more prone to this specific failure? Chapter 6 returns to exactly this question when it discusses what RLHF-style alignment does to a model's output distribution.

A methodological caveat, stated by the authors themselves

Before treating cosine similarity as a perfect stand-in for “are these two ideas actually the same,” hear the paper's own limitation section on this exact point: relying on semantic similarity of text embeddings to quantify diversity “may lack sufficient expressiveness to capture the full spectrum of creative variation in generated responses.” A cosine-similarity score is a useful, cheap, scalable proxy — it is not a perfect oracle for “these two responses express meaningfully different ideas.” Two responses could differ in ways an embedding model under-weights (a surprising structural choice, an unusual argument order) while still scoring similar, or differ in superficial wording while expressing the same core idea and scoring lower than they conceptually should. The random-pairs baseline above is what makes the headline numbers trustworthy in aggregate — it does not make every individual pairwise score a perfect semantic judgment.

See the caveat in real text. The appendix (Table 12) reports actual response pairs, with real similarity scores, for the query “create a short summary about the Nissan R390” — and one pair scored 0.803 despite directly contradicting each other on the car's actual specifications: one response describes “a twin-turbocharged 3.0-liter V6 engine… producing around 360–420 horsepower” with “only about 10–12 roadgoing examples,” while the other describes “a 3.5-liter V8 engine, producing around 650 horsepower” with a different, incompatible production history. These two responses cannot both be factually correct about the same car, yet they still score a healthy 0.803 — comfortably above the 0.8 “mode collapse” threshold used throughout this session — because cosine similarity over sentence embeddings is measuring topical and structural resemblance (same subject, same paragraph shape, same register), not factual consistency. This is exactly the boundary the paper's own limitation is warning about: a high similarity score means “these two responses occupy the same neighborhood of idea-space,” not “these two responses agree with each other,” and every number in this session should be read with that distinction in mind.

Compute it yourself: turning the histogram into one number

A histogram like the one above is ten separate percentages. It is useful to be able to collapse it into a single summary number, the same way a mean summarizes a whole distribution — and doing this derivation by hand, once, builds the exact skill you need to read any published histogram critically rather than just admiring its shape. Take the ten corpus-average buckets from Table 6 of the appendix and treat each bucket's midpoint (0.95 for the 0.9–1.0 bucket, 0.85 for 0.8–0.9, and so on down to 0.05) as a stand-in for every value inside it. A weighted average, weighting each midpoint by what fraction of the 2,500 (model, query) pairs fell into that bucket, estimates the overall mean similarity across the entire corpus-average distribution:

python
buckets = [(0.95,33.65), (0.85,36.47), (0.75,16.29), (0.65,7.46),
           (0.55,3.00), (0.45,1.54), (0.35,1.13), (0.25,0.43),
           (0.15,0.04), (0.05,0.00)]  # (midpoint, % of 2,500 pairs)
weighted_mean = sum(mid * pct for mid, pct in buckets) / 100
print(round(weighted_mean, 3))  # 0.829

An estimated mean of 0.829, sitting comfortably above the 0.8 threshold that 79% of flagship (model, query) pairs already exceed, is a useful sanity check on the whole histogram: it confirms the distribution is not just “mostly above 0.8 with a long low tail dragging the average down” — the bulk of the probability mass really is concentrated in the top two buckets, which is exactly what “mode collapse” as defined in Chapter 0 predicts you should see.

Run the identical weighted-mean calculation on Llama-3.2-1B-Instruct's bucket row from earlier in this chapter (5.00, 34.00, 38.00, 8.00, 5.00, 6.00, 1.00, 3.00, 0.00, 0.00) and the result is ≈0.739 — a full nine points lower than the corpus average's 0.829, and further still from GPT-4o's implied mean (51%×0.95 + 36%×0.85 + … ≈ 0.88). Three weighted-mean calculations, three different answers, one consistent ordering: smallest model < corpus average < largest flagship, exactly tracking the scale-versus-collapse pattern this chapter already flagged from the raw percentages. Deriving it yourself as a single summary number, rather than eyeballing ten-bucket histograms side by side, makes that ordering unambiguous.

Chapter 3: Turning Up the Temperature Doesn't Fix It

The obvious first fix, the moment you see 79%-of-cases-above-0.8, is: crank up the randomness. Standard top-p (nucleus) sampling at temperature = 1.0 already includes some randomness — surely turning the dial further would spread the model out across more of its answer space?

This is the single most common first instinct anyone encountering the mode-collapse result reaches for, and it is worth taking seriously precisely because it sounds so reasonable. Every one of the 25 flagship models in Chapter 2's histogram was already sampled with some randomness, at standard settings a production service would actually deploy — so the natural next experiment is not to ask a fundamentally different question, just to turn the same knob further and see whether the collapse was simply an artifact of under-exploring the model's own distribution.

Derived from zero: what temperature and top-p actually compute

Before judging whether turning up the randomness helps, be precise about what these two knobs actually do to the numbers. At every generation step, a language model outputs one raw score — a logit — per token in its vocabulary. The softmax function turns those logits into a probability distribution, and temperature T is a single scalar dividing every logit before that conversion:

At T = 1 you get the model's raw, as-trained distribution. As T grows past 1, dividing every logit by a bigger number squeezes the gaps between them before exponentiating, which flattens the resulting probabilities toward uniform — the model becomes less confident, more willing to pick a lower-scoring token. Top-p (nucleus) sampling then truncates that distribution: sort tokens by probability, keep adding tokens from highest to lowest until their cumulative probability first reaches threshold p, and sample only from that kept set (the “nucleus”), discarding everything else outright.

Work a small, illustrative example (five fictional candidate tokens, not from the paper) to see exactly how raising temperature changes the shape of the distribution top-p and min-p have to filter. Suppose the model's raw logits for the next token are [4.0, 3.0, 1.0, 0.5, 0.2]:

python
import math

def min_p_keep(logits, T, min_p):
    probs = [math.exp(z/T) for z in logits]
    total = sum(probs); probs = [p/total for p in probs]
    threshold = min_p * max(probs)
    return [p for p in probs if p >= threshold], threshold

logits = [4.0, 3.0, 1.0, 0.5, 0.2]  # illustrative, not from the paper
kept_t1, th1 = min_p_keep(logits, T=1.0, min_p=0.1)
kept_t2, th2 = min_p_keep(logits, T=2.0, min_p=0.1)
print(len(kept_t1), round(th1,3))  # 2 tokens, threshold 0.068 -- tightens when the model is confident
print(len(kept_t2), round(th2,3))  # 5 tokens, threshold 0.046 -- loosens as the distribution flattens

This toy calculation makes the paper's claim mechanically concrete: min-p's threshold is a fraction of the current top token's probability, not a fixed cutoff, so as temperature flattens the distribution and the top token's own probability drops, the absolute threshold drops with it, and more of the tail survives automatically — without needing a human to hand-tune a fixed nucleus size for every prompt. That self-adjusting property is exactly why the paper picks min-p, rather than simply cranking temperature alone under ordinary top-p, as its best attempt at a decoding-time fix.

What min-p sampling changes

The paper tests exactly this, using a decoding strategy called min-p sampling, published in a 2025 ICLR paper by Nguyen, Baker, Neo, Roush, Kirsch, and Shwartz-Ziv. Ordinary top-p sampling keeps the smallest set of tokens whose cumulative probability reaches a fixed threshold p, which at high temperature can let in a long tail of nearly-nonsensical low-probability tokens. Min-p instead sets a dynamic floor relative to the single most likely token's probability at each step — discard any token whose probability falls below min_p × P(most likely token) — so the cutoff tightens automatically when the model is confident and loosens automatically when it is uncertain. It is specifically designed to let you push temperature much higher than usual (the paper's own experiments go up to temperature = 2.0) while keeping the output readable.

The paper reruns the exact same 25-model, 100-query, 50-response-per-query protocol from Chapter 2, this time with top-p = 1.0, min-p = 0.1, temperature = 2.0 — about as aggressive a push toward randomness as you can make while the text stays coherent.

Why 50 samples, and not 5? A quick derivation from statistics

Both regimes generate 50 independent responses per (model, query) pair, not 5 or 10, and it is worth deriving why that number matters before trusting any comparison between them. If you sample a model n times and average the pairwise similarities among those samples, that average is itself an estimate of the model's true underlying similarity — and like any estimate built from a finite sample, it comes with uncertainty. The standard error of the mean shrinks proportionally to 1 ÷ √n: quadruple your sample size and your uncertainty shrinks by half, not by a quarter. Going from 5 samples to 50 samples (10× more data) only tightens your estimate by a factor of √10 ≈ 3.16 — diminishing, but real, returns.

Why this matters here specifically: the paper is comparing a baseline rate of 79% against a min-p rate of 61.2%, an 18-percentage-point gap. If each rate were estimated from only 5 samples per query instead of 50, the noise in each measurement could easily be large enough to make an 18-point gap statistically uninterpretable — you would not be able to tell whether min-p genuinely helps or whether you got a lucky/unlucky draw of 5 responses. Fifty samples per query, repeated across 100 queries and 25 models (2,500 independent (model, query) measurements feeding the histogram), is what makes an 18-point difference between two decoding regimes a signal worth trusting rather than sampling noise.

This is also why every chapter in this session keeps repeating exact sample counts (50 responses, 100 queries, 25 or 42 or 70+ models) rather than treating them as incidental methodology detail to skim past. Those counts are the difference between a percentage you can trust and a percentage you cannot.

The result: real improvement, nowhere near a fix

Read those two numbers together carefully. Cranking the sampling temperature to 2.0 — double the standard setting, about as aggressive as decoding gets before text degrades — buys you a drop from 79% to 61.2% in the fraction of cases landing above the 0.8 similarity threshold. That is a real, measurable improvement: roughly 18 percentage points of relief. But look at the 0.7 threshold: 81% of pairs are still above 0.7 similarity, which is itself far above the 0.1–0.2 random-pairs baseline from Chapter 2. Even the most aggressive diversity-oriented decoding strategy tested leaves the majority of the collapse intact.

Express the improvement as a relative, not just absolute, change to see how much of the original problem is actually gone: (79 − 61.2) ÷ 79 ≈ 22.5% relative reduction at the 0.8 threshold. Just under a quarter of the collapse, as measured at that specific cutoff, goes away under the most aggressive diversity-oriented decoding setup tested — real, worth having, and still leaving more than three-quarters of the original 79% figure standing. Whichever way the two numbers get framed — 18 raw percentage points, or 22.5% relative — neither framing supports treating decoding-time randomness as a solution rather than a partial mitigation.

Why decoding-time randomness cannot fully solve a distribution-shape problem

Here is the mechanism, and it is worth deriving rather than just accepting. Sampling temperature and min-p both operate at generation time, reshaping how aggressively you sample from the probability distribution the model already computed for the current token. They can widen which individual tokens get selected at each step — more synonyms, more sentence-level variation, different word order. What they cannot do is change the underlying shape of the distribution itself: if the model's learned distribution over entire responses already has 90% of its mass sitting on river-metaphor-shaped continuations and 8% on weaver-metaphor-shaped continuations, no amount of per-token temperature tweaking invents a third, tiger-shaped continuation the model never learned to place meaningful probability on. You can reshuffle word choice within a mode. You cannot decode your way into a mode that was never there.

There is also a practical adoption problem the paper is candid about: min-p is well-suited to creative, open-ended generation, but performs worse on closed-ended tasks where you actually want the model to converge on one correct answer — meaning a service cannot simply flip it on globally without hurting the majority of its non-creative traffic. And relying on any decoding-level fix at all quietly shifts the burden onto individual users to know which knob to turn, rather than the model behaving well by default.

Why min-p needs top-p turned off, but not turned reckless

One detail in the min-p configuration is easy to skim past: top-p = 1.0. Setting top-p all the way to 1.0 does not mean “use both filters” — it means the nucleus filter is switched off entirely (a cumulative-probability cutoff of 1.0 always includes every token), leaving min-p as the only thing standing between the raw, temperature-flattened distribution and total incoherence. Extend the five-token toy example from earlier by adding one more, deliberately terrible candidate — a sixth, near-nonsensical token with a very low raw logit of −2.0 — and check what each filter alone would do to it at T = 2.0:

python
import math

logits = [4.0, 3.0, 1.0, 0.5, 0.2, -2.0]  # added a 6th, near-garbage token
T = 2.0
probs = [math.exp(z/T) for z in logits]
probs = [p / sum(probs) for p in probs]
print([round(p, 4) for p in probs])
# [0.454, 0.2753, 0.1013, 0.0789, 0.0679, 0.0226] -- garbage token still gets ~2.3% probability

# top-p=1.0 alone: keeps ALL 6 tokens, garbage included, at full 2.26% sampling odds
# min-p=0.1 alone: threshold = 0.1 * 0.454 = 0.0454 -- garbage token's 0.0226 falls BELOW it, discarded

With top-p set to 1.0 and no min-p filter at all, that garbage token would still get sampled roughly 1 time in 44 — not rare enough to ignore across 2,048 tokens of generation. Min-p's dynamic threshold, scaled to whatever the current top token's confidence happens to be, catches and discards it while still admitting the four other reasonable candidates that top-p=0.9 at ordinary temperature would have excluded. This is the precise engineering reason the paper pairs top-p = 1.0 with min-p = 0.1 rather than simply disabling all filtering at high temperature: min-p is doing real work as a coherence floor, even while it is letting far more of the distribution's reasonable tail through than the baseline configuration did.

This is a small but real example of a pattern that shows up throughout responsible ML engineering: a fix for one problem (coherence collapsing at high temperature) needs to be paired with a fix for the problem the first fix would otherwise reintroduce (garbage tokens re-entering once top-p stops doing any filtering). Neither knob alone gets you where you want to be; the combination does specific, complementary work, and understanding the mechanism of each — rather than treating “turn up the sampling settings” as one undifferentiated action — is what let this chapter separate “real, bounded improvement” from “the collapse problem solved.”

An honest scoping note: this chapter is about diversity, not quality

One more caveat, stated directly in the paper's own limitations section, matters for how much weight to put on every number in this chapter: the study “focused primarily on diversity, with fixed model decoding configurations, without studying quality” as a separately measured axis. The authors selected sampling settings that produce “empirically coherent text” and then asked whether models can be pushed toward more diverse output while staying in that coherent regime — but nothing in this chapter's 61.2%-versus-79% comparison is a quality score. A wilder, more diverse-looking response is not automatically a better one, and a service deciding whether to adopt min-p in production would still need a separate quality evaluation on top of everything measured here. Diversity and quality are two different dials, and this entire session studies only one of them.

Carry one open engineering question forward into Chapter 4: this chapter only tested varying how one model samples from its own distribution. It never asked whether a completely different model, sampled at ordinary settings, might land somewhere genuinely different in idea-space. That is the next escape hatch worth closing — and Chapter 3's own result (temperature can reshuffle phrasing within a mode but not manufacture a new one) makes a specific, testable prediction about what Chapter 4 should find: if different models were trained on genuinely different data with genuinely different priorities, their modes should differ, even where single-model decoding tricks cannot help. Whether that prediction survives contact with real data is exactly what comes next.

Chapter 4: Different Models, Same Answer

Chapter 3 established that one model cannot decode its way out of repeating itself. This chapter asks the more unsettling question: if you can't trust one model to be diverse, can you at least trust different models — built by competing companies, on different data, with different architectures — to diverge from each other?

Set an expectation before looking at the answer, the same way Chapter 0 did for the intra-model case. If two models were trained fully independently — different pretraining corpora, different alignment teams, different reward models, no shared engineering decisions — there is no obvious mechanical reason their outputs on an open-ended prompt should land anywhere near each other beyond the ordinary topical overlap any two competent models would share. A reasonable naive prior might put independent-model similarity somewhere between the random-pairs floor (0.1–0.2) and same-model similarity (0.8-plus) — genuinely different, but not random. Chapter 4 tests that prior directly.

The numbers

Using the same 1,250-responses-per-query pool from Chapter 2, the paper now compares responses across models instead of within one model. Average pairwise similarity between different models' responses to the same query ranges from 71% to 82%, depending on which pair of models you look at. Two concrete examples the paper reports by name: DeepSeek-V3 and qwen-max-2025-01-25 average 0.82 similarity to each other; DeepSeek-V3 and gpt-4o-2024-11-20 average 0.81. These are four organizations — a Chinese AI lab, Alibaba, and OpenAI — whose models were trained independently, on different infrastructure, with different alignment recipes, converging on open-ended answers about as tightly as GPT-4o converges on itself across two different snapshot dates (recall Chapter 2: same-model similarity typically sits in the low-to-high 0.8s).

Sit with that comparison for a second, because it is the sharpest single sentence in this whole session. GPT-4o comparing itself to itself, one snapshot to another, sits in the low-to-high 0.8s. DeepSeek-V3, a model built from scratch by an entirely different organization, sits at 0.81 against GPT-4o and 0.82 against Qwen-Max. The “are these the same model or different models” question, answered purely from the similarity number alone with no other information, is close to a coin flip. That is the concrete, numeric meaning behind this chapter's title: different models, same answer.

The paper flags something specific worth naming: OpenAI's models and Qwen's API models show unusually high cross-similarity even relative to other pairings, and the authors are honest that they cannot fully explain why — candidate explanations include overlapping data pipelines or contamination from shared synthetic data, but the paper stops short of claiming a proven cause. This is presented as an open question for future work, not a solved mystery.

Notice how carefully hedged that sentence is, and treat the hedging as a model for how to read the rest of this chapter. “Overlapping data pipelines or contamination from shared synthetic data” are two candidate explanations, offered side by side, neither singled out as confirmed. This is exactly the right level of confidence for an observation this specific and this hard to verify from outside — training-data pipelines are proprietary, and no amount of output-level analysis alone can fully distinguish “these two companies' models happen to share upstream synthetic-data suppliers” from “these two companies converged independently on similar alignment recipes that happen to produce similar typicality biases” (the mechanism Chapter 6 builds). Both are plausible. Only one, if either, is true. The paper does not pretend to know which.

Reading the full similarity matrix: what GPT-4o's own row looks like

The 71%–82% range quoted above is a summary of a much larger object: the paper's appendix (Tables 7–11) reports a full model × model matrix of average pairwise similarity, one cell for every pair among the 25 main-paper models, computed the same way as Chapter 2's intra-model number but comparing responses from two different models to the same query instead of fifty responses from one. The matrix's diagonal — a model compared against itself — reproduces Chapter 2's intra-model averages almost exactly (gpt-4o-2024-11-20 against itself: 88.2%, consistent with its 51%+36%=87% above-0.8 rate from Chapter 2). Reading across one full row of that matrix makes the “how different is different, really” question concrete:

Two things pop out of one real row. First, DeepSeek-V3 — a completely independent company's model — ties gpt-4o-mini, OpenAI's own smaller sibling model, for GPT-4o's second-highest similarity score in this table, at 81.4%. Second, even GPT-4o's single least similar pairing in this row — against the smallest, weakest model tested, Llama-3.2-1B-Instruct — still sits at 70.1%, which is roughly seven times higher than where the random-pairs baseline from Chapter 2 lives (0.1–0.2, or 10–20%). There is no pairing anywhere in this row that looks like genuine independence. The floor of “how different two arbitrary frontier models can be” on this benchmark is still a long way from “actually different.”

There is a practical engineering implication worth drawing out (this lesson's own inference from the numbers, not a paper claim): a “model router” that falls back from one provider to another during an outage — treating the swap as invisible to the end user — is, on the evidence in this row, a much safer bet for open-ended creative traffic than a naive reading of “these are completely different companies' models” might suggest. The output a user gets back after a silent failover is unlikely to feel jarringly different. That consistency is convenient for infrastructure reliability. It is the exact same fact that makes building genuine redundancy or diverse perspective out of a multi-provider setup much harder than it looks — the same number that reassures an SRE is the number that should worry a team assuming their model swarm is sampling from meaningfully different corners of idea-space.

When it's not just similar — it's identical

Averages can hide how extreme the tail looks. Two concrete instance-level examples make the severity vivid.

Verbatim phrase overlap. For the prompt “Create a description with 2-3 sentences for an iPhone case collection that is a slim-fitted case with bold designs,” DeepSeek-V3 and gpt-4o-2024-11-20 — models from two unrelated companies — both produced the phrases “Elevate your iPhone with our,” “sleek, without compromising,” and “with bold, eye-catching” in their responses. Not the same idea in different words — the same words.

Byte-identical output. For the prompt “Generate a motto for a social media page focused on successes, wealth, and self-help,” both qwen-max-2025-01-25 and qwen-plus-2025-01-25 generated the exact same string, character for character: “Empower Your Journey: Unlock Success, Build Wealth, Transform Yourself.” Two different model sizes, same company, same output — not similar, identical.

The flagship example, quantified

Return to Chapter 0's opening scenario with the actual measurement method behind it. For the prompt “write a metaphor about time,” the paper takes all 25 models' 50 responses each (1,250 responses total), embeds every one with text-embedding-3-small, and applies PCA — a dimensionality reduction technique that projects those 1,536-dimensional embedding vectors down onto the two directions that capture the most variation, so the whole cloud of 1,250 points can be plotted on a flat page. (If PCA is new to you, it is the same family of technique used any time a high-dimensional embedding space gets visualized as a 2D scatter plot — another direct link back to this course's embedding sessions.) The result: not a diffuse cloud, but two tight clusters — a dominant one anchored on “time is a river,” a smaller one on “time is a weaver” — and critically, both clusters are populated by dots from many different models, not segregated by which model produced them.

What a 0.86 versus a 0.59 actually reads like

It is easy to let cosine-similarity numbers become abstract. The paper's appendix (Table 16) reports real response pairs for this exact query, “write a metaphor involving time,” with their exact similarity scores, and reading two of them side by side is the fastest way to trust that the two-cluster result is a real semantic split, not a quirk of the embedding metric. A high-similarity, same-cluster pair, scored 0.864:

Now a lower-similarity, cross-cluster pair — one river response, one weaver response, both real answers to the identical prompt — scored 0.592:

0.864 versus 0.592 — a gap of more than a quarter on the 0–1 scale, and it tracks exactly what a human reader would call the difference: two river metaphors, however differently worded, share the whole governing image (flowing, currents, navigating); a river metaphor and a weaver metaphor do not, no matter how similar their sentence-level craftsmanship is. This is the concrete, readable version of Chapter 4's abstract PCA claim — the embedding space is not randomly scattering these responses, it is placing river-family responses near each other and weaver-family responses in a genuinely different neighborhood, which is exactly what makes “two tight clusters, not one diffuse cloud” a trustworthy visual summary rather than an artifact of how PCA happened to draw the plot.

The most striking single statistic: how many distinct models share one cluster

The paper quantifies this mixing directly. For each query, take the top-N most mutually similar responses across the entire 1,250-response pool and count how many distinct models contributed to that set. If every model were perfectly disjoint from every other — total inter-model diversity — the top-50 most-similar cluster would be dominated by one model's own internal repeats. If models were maximally spread across each other, that cluster could in principle draw from up to all 25.

Eight is a genuinely striking middle ground. It is far more than the “1” you'd see if every model kept purely to itself, which means real cross-model contamination of ideas is happening. It also means that, of the fifty most alike responses to any given open-ended question in this study, on average a third of all the model families tested are each contributing near-duplicates to that one cluster. The paper notes some queries exceed 10 distinct models in a single top-50 cluster — and that in some cases, two different models turn out to be more similar to each other than a single model is to its own resampled self.

How you would actually compute this yourself

The paper's description — “identify the top-N most similar outputs for each query and count the unique models contributing” — leaves the exact algorithm implicit. Working through one reasonable way to implement it, on a tiny toy pool, makes the statistic concrete rather than abstract. Start from the full 1,250-response pool for one query (25 models × 50 responses), embed every response, and for each response compute its average similarity to every other response in the pool — a single number per response summarizing “how typical is this response of the whole crowd.” Sort all 1,250 responses by that number, descending, and take the top 50: those are the 50 responses that are, on average, most similar to everything else in the pool — the “most typical” cluster. Then simply count how many distinct model-IDs appear among those top 50.

python
# illustrative implementation sketch, toy scale (6 responses from 3 models, not paper data)
responses = [
    {'model': 'A', 'emb': emb_a1}, {'model': 'A', 'emb': emb_a2},
    {'model': 'B', 'emb': emb_b1}, {'model': 'B', 'emb': emb_b2},
    {'model': 'C', 'emb': emb_c1}, {'model': 'C', 'emb': emb_c2},
]
for r in responses:
    sims = [cosine_similarity(r['emb'], other['emb'])
            for other in responses if other is not r]
    r['avg_sim'] = sum(sims) / len(sims)   # "how typical is this response"

top_n = sorted(responses, key=lambda r: -r['avg_sim'])[:4]  # top-4, out of 6, for this toy example
distinct_models = len(set(r['model'] for r in top_n))
print(distinct_models)  # if this returns 3 (out of 3 possible), the "most typical" cluster is fully mixed across models

Scale that exact idea up from 6 toy responses and 3 toy models to the paper's real 1,250 responses and 25 real models, and “distinct_models” on the top-50 slice is precisely what averages to ≈8 across queries. The statistic is not measuring how similar any one pair of models is — it is measuring whether the crowd's most-agreed-upon, most-typical answers happen to be drawn from many different sources or just a few, which is exactly the property an ensemble or model swarm is implicitly betting on when it assumes “more models sampled = more independent perspectives represented.”

Chapter 5: Paraphrase Doesn't Save You

One more escape hatch is worth closing before moving on to why this happens. Maybe the convergence observed so far is really just surface-level phrase memorization — models latching onto the exact wording of a specific prompt they've seen many times in training data, so that rephrasing the question would break the spell and let genuinely different ideas through.

The experiment

The paper tests this directly. Starting from 30 prompts drawn from Infinity-Chat100, gpt-4.1-2025-04-14 generates 4 paraphrases of each — reworded, same intent, verified by a separate LLM-judge pass to confirm the meaning didn't drift. That gives 30 × (1 original + 4 paraphrases) = 150 distinct prompt strings. Each of 42 representative models then generates 20 responses to every one of those 150 prompts, at the by-now-familiar top-p = 0.9, temperature = 1.0.

For every model, the paper computes two numbers: within-prompt similarity — average pairwise similarity among the 20 responses to the exact same original prompt — and cross-paraphrase similarity — average similarity between responses to the original prompt and responses to its four differently-worded paraphrases. If phrase memorization were the whole story, cross-paraphrase similarity should drop sharply relative to within-prompt similarity, because the model would no longer be pattern-matching against one specific memorized string.

Notice the scale jump from Chapter 2's protocol: this robustness check runs 42 models, not the 25 flagship models used for the main intra- and inter-model figures. That is a genuine expansion, not a subset — drawn from the same 70+-model roster Chapter 1 introduced, it pulls in additional checkpoints across the Qwen, Llama, and Gemma families beyond just each family's single strongest representative, specifically so the paraphrase-robustness finding cannot be dismissed as an artifact of studying only the very largest, most heavily-aligned models. If the “profound” template survives across 42 differently-sized checkpoints, not just the 25 flagships, that is stronger evidence the pattern reflects something general about how these models are trained, not a quirk specific to whichever handful of models happened to be in the main-text figures.

Why the paraphrases needed their own verification step

Notice the extra safeguard baked into the experiment's design: paraphrases were not just generated and used as-is, they were passed through “further LLM judge verification” to confirm meaning did not drift. This matters because a sloppier version of this experiment could accidentally undermine its own conclusion. If a “paraphrase” secretly asked a subtly different question — not just different words, but a genuinely different intent — then a drop in cross-paraphrase similarity would prove nothing about memorization; it would just mean the two prompts weren't actually asking for the same thing anymore. By verifying semantic equivalence before measuring similarity, the paper protects the experiment's own logic: any similarity that survives is guaranteed to be measuring “does rewording alone break the spell,” not “did the rewording accidentally change the question.” This is the same discipline as Chapter 2's random-pairs baseline — a control designed specifically to rule out the most obvious alternative explanation before trusting the headline number.

It also means the 0.821-versus-0.781 comparison is, if anything, biased slightly against finding a small gap, not toward it. Any paraphrase that drifted even slightly off the original intent would have been caught and presumably discarded or corrected before the response-generation step, which removes exactly the subset of paraphrases most likely to produce genuinely different responses for a genuinely different reason. A sloppier experiment, one that let some intent-drifted “paraphrases” slip through, would likely have shown a larger apparent recovery from rewording — and that larger number would have been partly spurious, an artifact of secretly changing the question rather than evidence that rewording defeats memorization. The verification step makes the small 4.9% figure this chapter reports a more conservative, more trustworthy estimate of what pure rewording alone can buy back.

The result

Two numbers to hold side by side, computed from that 126,000-response pool.

Rewording the question — changing every word while carefully preserving the intent — buys back only about 4.9% of similarity, relatively speaking. Responses to “write a metaphor about time” and responses to a completely differently-phrased paraphrase of the same request are still, on average, over 0.78 cosine-similar to each other. This is the same magnitude of gap Chapter 0 found for the answer-space-size comparison (4.4%) — a recurring pattern in this paper: interventions that sound like they should matter a lot (a bigger answer space, a differently worded question) move the needle by only a few percentage points.

Put the two 42-model averages on the same 0-to-1 scale this session has used throughout, and the visual gap between “same exact prompt” and “fully reworded prompt” is barely wider than the width of the line separating them: 0.821 sits just above 0.8, and 0.781 sits just below it. Both numbers are comfortably inside the “collapsed” regime Chapter 2 defined, on either side of the rewording. A gap this small, on a scale where the true floor (random, unrelated responses) sits down near 0.15, is the numeric definition of “paraphrasing barely moved anything.”

One model, five genuinely different sentences, one river

Percentages can feel abstract. The paper's appendix (Table 17) makes the same point with an unedited transcript from a single real model, Llama-3.1-8B-Instruct, answering five differently-worded prompts, all asking for the same thing:

Read those five left-column prompts again: no two of them share more than a word or two in common (“metaphor,” “time”), and a gpt-4.1-2025-04-14 judge pass confirmed they all genuinely preserve the same request. Yet the right-hand column is the same idea, five times, with only the sentence's connective tissue changing. This single transcript is the 0.821-versus-0.781 gap made visible: the model is not retrieving a memorized string keyed to specific input tokens — it is retrieving a concept (“time → river”) keyed to an intent, and every rewording still triggers that same intent classification.

A second instance, showing the pattern is structural, not lexical

A different prompt and model make the same point in a different way. Table 18 of the appendix shows Llama-3.3-70B-Instruct answering five paraphrases of “write a paragraph about how the internet shaped society,” and this time the repeated element is not a single noun (like “river”) but an entire sentence template: every one of the five responses opens with some variant of “the internet has profoundly [reshaped/impacted/revolutionized] society,” and every one continues into some ordering of “transforming/revolutionizing the way we communicate, access information, and interact with one another” — not identical words every time, but the identical slot structure, filled in with near-synonyms, five separate times, for five prompts that share almost no surface wording with each other. This is the clearest evidence in the whole session that what gets memorized during training is not a string, and not even quite a single concept, but something closer to a template: a fixed shape of sentence the model has learned is the “correct-sounding” way to open a response to this entire family of requests.

The same table shows this is not one model's idiosyncrasy. Two entirely separate model families — Qwen2.5-72B-Instruct (“the internet has profoundly reshaped society in numerous ways…”) and gemma-2-27b-it (“the internet has profoundly reshaped society…”) — independently reach for the identical opening word, “profoundly,” across their own five paraphrased prompts, with no shared training run and no shared company connecting them to Llama-3.3-70B-Instruct. This is Chapter 4's inter-model convergence and this chapter's paraphrase-robustness finding, stacked on top of each other in one real example: three unrelated model families, five differently-worded prompts each, and the surface template survives both axes of variation at once.

There is a direct practical consequence for anyone building a synthetic prompt-diversification pipeline — a common technique where an engineer generates many paraphrases of a seed instruction to expand a training set's apparent variety. This chapter's result says that technique diversifies the questions without reliably diversifying the answers. If the goal is genuinely varied training examples, paraphrasing the prompt side of the pipeline is not sufficient on its own; the responses still need to be checked for the same kind of template-level convergence this chapter just measured directly, using the exact embed-and-compare protocol Chapter 2 introduced.

What this rules out, and what it implies

This result closes the phrase-memorization loophole cleanly: the convergence documented in Chapters 2 through 4 is not the model pattern-matching on one exact string it has memorized a canned answer for. It survives genuine semantic rewording almost fully intact. That means the collapse lives in concept space, not string space — the model has learned that a certain family of ideas is the “expected” response to a certain family of intents, and it retrieves that family of ideas regardless of the exact words used to ask for it.

Push on the “intent” framing for a moment, because it is doing real conceptual work. An intent classifier — whether hand-built or implicitly learned inside a language model's own weights — maps a wide variety of surface phrasings onto a much smaller set of underlying categories. That is normally considered a feature, not a bug: it is exactly what lets a customer-support chatbot correctly route “I want a refund,” “can I get my money back,” and “this doesn't work, undo the charge” to the same handling logic, regardless of which words a particular user happened to reach for. This chapter's finding says that same collapsing-many-phrasings-to-one-bucket behavior, useful for routing requests, is also operating one level up — not just on which handler processes the request, but on which family of ideas gets retrieved to answer it. The mechanism that makes intent classification robust is structurally the same mechanism producing the narrowness this whole session documents.

Placing the observed gap between two extreme hypotheses

Turn the intuition above into an actual measurement scale. Define two extreme hypotheses for what “paraphrasing” should do to similarity, and see which one the real 0.821-versus-0.781 numbers sit closer to.

Pure string memorization (H1): if a response is effectively a lookup keyed to the literal prompt text, changing every word should destroy the key entirely — cross-paraphrase similarity should collapse toward whatever the paper's random-pairs baseline established for genuinely unrelated responses, roughly the 0.10–0.20 range's midpoint, ≈0.15. Pure concept-level retrieval (H2): if a response is keyed only to the semantic intent, and paraphrasing perfectly preserves intent, cross-paraphrase similarity should be statistically indistinguishable from within-prompt similarity — the gap should be essentially zero.

python
within_sim, cross_sim = 0.821, 0.781
random_baseline_center = 0.15   # midpoint of the paper's 0.10-0.20 random-pairs range

gap_observed = within_sim - cross_sim
gap_if_pure_memorization = within_sim - random_baseline_center

position = gap_observed / gap_if_pure_memorization
print(round(position, 3))   # 0.060 -- 94% of the way toward "concept-level," only 6% toward "string memorization"

Ninety-four percent of the way toward pure concept-level retrieval, and only six percent of the way toward pure string memorization — on a scale this lesson built specifically to make that judgment call explicit rather than a matter of prose intuition. Note the honesty boundary here too: the H1/H2 framing and the 0.15 baseline midpoint are this lesson's own analytical tool for interpreting the paper's real numbers, not a calculation the paper itself performs. The two input numbers — 0.821 and 0.781 — are exactly what the paper reports.

This leaves one question hanging over the whole session so far: if it isn't the answer space (Chapter 0), isn't fixable by decoding (Chapter 3), isn't confined to any one model (Chapter 4), and isn't phrase memorization (this chapter) — where, mechanically, does the narrowing actually get baked in? Chapter 6 goes looking for an answer in the one place none of the previous five chapters examined directly: not what the model does at inference time, but what happens to it during training.

Chapter 6: Why Modes Form: The Typicality Hypothesis

Every chapter so far has measured that collapse happens and ruled out several tempting explanations for it — the answer space isn't actually smaller than it looks (Ch 0), decoding-time randomness can't fix it (Ch 3), and it isn't phrase memorization (Ch 5). None of that yet explains why a model trained on an enormous, genuinely diverse pretraining corpus ends up narrow at inference time. This chapter builds the leading explanation from the wider alignment literature — and it is important to be precise about what is established fact from this paper versus informed hypothesis from elsewhere, because the paper itself is candid that it has not run the controlled experiment that would prove this mechanism directly.

Frame the question precisely before reaching for an answer. Pretraining corpora are enormous and genuinely varied — billions of documents, written by millions of different people, in every register and style imaginable. If a model's final behavior simply reflected what it saw during pretraining, in roughly the proportions it saw it, open-ended prompts should produce something close to that same variety. They do not. Something between raw pretraining and the model a user actually talks to is doing the narrowing, and this chapter's entire job is to locate, as precisely as the evidence allows, where in that pipeline the narrowing happens.

What the paper actually says about this

In its own stated future directions, the authors write that they intend to “quantify the relative contributions of different post-training pipelines — such as supervised fine-tuning, RLHF/RLAIF, and constitutional training — to the emergence of homogenized responses.” That is an open research question the paper flags, not a result it reports. What it does offer as suggestive, indirect evidence is a citation to a 2025 paper by Peter West and Christopher Potts, titled, tellingly, “Base models beat aligned models at randomness and creativity.” The title alone states the direction the field's evidence points: pretrained base models — before any instruction-tuning or preference optimization — tend to be more diverse than their aligned counterparts, the same models after RLHF or a similar post-training pipeline has been applied.

The mechanism, built from first principles

Here is why that pattern would be expected, reasoned from how these training stages actually work. A base model's pretraining objective is, roughly, to match the probability distribution of its training corpus — the internet's genuinely wide range of ways people write about any given topic. That is a mode-covering objective: it wants probability mass spread across every pattern that shows up often enough in the data, including many different metaphors for time, many different brainstorming angles, many different writing styles.

Reinforcement learning from human feedback (RLHF) works differently. A reward model is trained on human pairwise comparisons — given two responses, which one do annotators prefer — and then the base model is fine-tuned to maximize that reward, usually with a KL-divergence penalty holding it close to its starting distribution (this course's DPO session, Session 06, derives that KL “leash” term in full if you want the underlying math). Optimizing hard against a scalar reward signal is, structurally, a mode-seeking process: rather than spreading probability across every response humans rated “pretty good,” it concentrates probability onto whichever region of response-space scores highest under the reward model, because that is exactly what the optimization is rewarded for doing. Pushed far enough, mode-seeking optimization does not sample proportionally from a rich distribution of good answers — it collapses onto the single best-scoring peak it can find and stays there.

Write the objective the RLHF stage actually optimizes, in full, to see exactly where the mode-seeking pressure enters the math:

The first term is the expected reward the current policy πθ earns under the learned reward model rφ — this is the term doing the mode-seeking work, because gradient ascent on an expectation of a scalar reward has no built-in reason to preserve probability mass on a second, lower-scoring-but-still-valid mode; it simply pushes weight toward whatever scores higher. The second term, weighted by β, is the KL penalty holding the policy close to the reference (typically the SFT model before RLHF) — this is the only force in the whole objective actively resisting collapse. Turn β up and the model stays closer to its more mode-covering starting distribution, sacrificing some of the reward gain; turn β down and the reward term dominates, and the policy is freer to collapse onto whatever the reward model scores highest, exactly the dynamic that produces a mode-seeking outcome. Chapter 6's typicality-bias hypothesis is, in this formalization, a claim about rφ itself: if the reward model systematically scores the “typical” response even slightly higher than an equally-valid atypical one, then any nonzero amount of reward-term optimization pressure — regardless of how carefully β is tuned — pushes probability mass away from the atypical mode, a little more with every training step.

Reason through the two extremes of β to see why there is no setting that eliminates the problem outright, only trades it off. At β → 0, the KL penalty vanishes entirely and the objective reduces to pure reward maximization — the policy is free to walk arbitrarily far from its mode-covering starting point, and in the limit collapses onto whatever single response (or narrow family of responses) the reward model scores highest, exactly the “time is a river, always” failure mode this whole session documents. At the opposite extreme, β → ∞, the KL term dominates so completely that any deviation from the reference policy incurs unbounded penalty — the model simply never moves, earning zero reward-model-driven improvement at all, uninteresting for a different reason (you would not be doing RLHF). Every real deployment picks a finite β in between, and that choice is precisely a diversity-versus-alignment-strength trade-off: a smaller β lets the reward model reshape behavior more aggressively (better at what the reward model measures, worse at retaining modes it does not reward), a larger β preserves more of the base model's original spread at the cost of weaker alignment to the reward signal. Nothing about tuning β alone fixes the underlying problem Chapter 6 raises — it only chooses how much of it you are willing to accept in exchange for how much alignment strength you get. That is exactly why the paper's own future-work section proposes new objectives, not just new hyperparameter settings, as the mitigation worth exploring.

Derived from zero: why "mode-covering" and "mode-seeking" are not just labels

Those two words are not loose metaphors — they describe a real, measurable asymmetry in how a fitting objective reacts to a true distribution with more than one good answer. The tool that makes this precise is KL divergence, KL(A || B) = ∑i Ai · log(Ai ÷ Bi), which measures how many extra “nats” of surprise you incur using distribution B to describe outcomes that are actually drawn from distribution A. The crucial asymmetry: KL(A||B) ≠ KL(B||A) in general, and which one gets minimized determines completely different behavior when the true distribution has more than one peak.

Work a small, honest, hand-computable toy example (four buckets, illustrative numbers, not paper data). Let the true, human-judged distribution of good ideas for some open-ended prompt be P = [0.45, 0.35, 0.12, 0.08] (river, weaver, thief, sculptor — two real dominant ideas, two minor ones). Compare two candidate model distributions: a balanced approximation that roughly tracks P, QA = [0.40, 0.35, 0.15, 0.10], and a collapsed approximation that nearly abandons everything except the dominant idea, QB = [0.97, 0.01, 0.01, 0.01].

python
import math

P  = [0.45, 0.35, 0.12, 0.08]   # illustrative "true" distribution of good ideas, not paper data
QA = [0.40, 0.35, 0.15, 0.10]   # balanced approximation
QB = [0.97, 0.01, 0.01, 0.01]   # collapsed onto one idea

def kl(A, B):
    return sum(a * math.log(a / b) for a, b in zip(A, B))

print(round(kl(P, QA), 3))   # forward KL(P||QA)  = 0.008 nats -- barely any surprise
print(round(kl(P, QB), 3))   # forward KL(P||QB)  = 1.363 nats -- huge penalty
print(round(kl(QA, P), 3))   # reverse KL(QA||P)  = 0.009 nats
print(round(kl(QB, P), 3))   # reverse KL(QB||P)  = 0.664 nats -- large, but far less than forward KL's 1.363

Read the two “collapsed” numbers side by side: forward KL punishes Q_B's neglect of the real 0.35-probability “weaver” idea with a penalty of 1.363 nats — more than double what reverse KL charges the same collapsed distribution, 0.664 nats. That gap is the entire mechanism in one number: forward KL, KL(Ptrue||Qmodel), is what next-token pretraining effectively minimizes (it is the same quantity as the cross-entropy loss, up to a constant that doesn't depend on the model), and it punishes a model harshly for assigning near-zero probability to any idea that genuinely occurs in the data — which is exactly why a pretrained base model, trained this way over an enormous, genuinely diverse corpus, keeps probability mass spread across many plausible continuations. Reward-model optimization, by contrast, behaves much more like minimizing reverse KL, KL(Qmodel||Ptrue) — a criterion that is comparatively forgiving about abandoning a real mode entirely, as long as the model doesn't waste probability mass on regions the reward signal scores badly. That asymmetry, not a vague appeal to “RLHF makes models boring,” is the precise mathematical reason mode-seeking optimization can walk away from a real, human-validated secondary idea and pay a smaller price for doing so than a mode-covering objective ever would.

One honest technical caveat on this derivation: real RLHF training is not literally computing or minimizing an exact reverse-KL term the way the four-bucket toy example above does — the actual mechanics involve policy-gradient estimates on sampled rollouts, not a closed-form divergence over an explicit probability table. The toy calculation's purpose is narrower and more honest than “this is exactly what RLHF computes”: it demonstrates that an optimization objective built around maximizing expected reward, restrained only by a KL penalty toward a reference policy, has the qualitative asymmetry this section describes — comparatively more tolerant of abandoning a true secondary mode than an objective that instead minimizes forward KL against the true data distribution. That qualitative asymmetry, not this specific four-number toy example, is the actual mechanism being argued for.

Naming the effect: typicality bias

Put a name on the specific failure this predicts: typicality bias. If human annotators — and the reward models trained to imitate their preferences — systematically rate the most expected, safest, most “normal-sounding” response as marginally better than an equally valid but less common alternative (which is a very natural bias for annotators working quickly through thousands of comparisons to have), then optimizing against that signal squeezes probability mass toward the typical answer at every opportunity, generation after generation of training. Nothing about the less-typical alternative was actually worse. It just consistently loses the marginal comparison that decides what gets reinforced.

Why would annotators lean this way, mechanically, rather than randomly? Comparing two open-ended responses is cognitively harder than comparing two closed-ended ones, because there is no ground truth to check against — the annotator has to fall back on their own sense of what a “good” answer to this kind of question usually looks like. A response that matches that prior expectation requires less scrutiny to approve; a response that violates it requires the annotator to stop and actively justify why something unusual is still good, which working quickly through thousands of comparisons gives little incentive to do. The bias does not need any annotator to be careless or lazy in isolation — it can emerge purely from the asymmetry between how much cognitive effort “matches my prior” and “surprises me but is still valid” respectively demand, aggregated across many fast judgment calls.

This also offers a candidate explanation for the OpenAI/Qwen cross-similarity anomaly flagged in Chapter 4: if independently-trained models are all being pushed toward whatever counts as “typical and safe” by broadly similar alignment methodologies — RLHF-style preference optimization, trained on human comparisons that likely share similar typicality biases across different labeling pipelines — then convergent training processes, not shared weights, would be sufficient to produce convergent outputs. The paper observes the correlation (companies with more similar alignment approaches tend to produce more similar outputs); it does not claim to have proven this specific causal chain. Hold this chapter's mechanism as the leading, well-motivated hypothesis the field currently has — and as exactly the kind of controlled ablation (same base model, vary only the post-training recipe) the paper's own future-work section calls for.

The paper's full future-work plan for testing this hypothesis

It is worth reading the paper's own future-directions section in full rather than the single sentence already quoted above, because it lays out a concrete, three-part research program — and being explicit about what is planned versus what has been run reinforces exactly how much of this chapter is hypothesis.

Foundation and training analysis extends the Artificial Hivemind testbed to “foundation models without instruction-following capabilities” — base models, before any post-training at all — specifically to disentangle how much of the narrowing comes from pretraining itself versus everything that happens afterward, and to separately measure the individual contributions of SFT, RLHF/RLAIF, and constitutional training rather than lumping “alignment” into one undifferentiated stage. Mitigation and alignment strategies proposes exploring “diversity-aware training objectives and alignment schemes that explicitly reward exploration of multiple valid modes while preserving response quality,” plus benchmarking decoding strategies this paper did not test — diverse beam search and other nucleus-sampling variants — under the same Artificial Hivemind measurement framework Chapter 2 built. Practical integration proposes folding the framework into red-teaming workflows (to stress-test whether a model's coverage of an open-ended prompt space is adequate), using Infinity-Chat as a training-prompt resource for reinforcement-learning methods that explicitly reward diversity, and using the diagnostic itself to inform curriculum design — gradually exposing a model to the open-ended prompts most susceptible to collapse during training.

None of these three research directions has been executed in the paper this session is built on. They are the authors' own explicit roadmap for turning Chapter 6's hypothesis into a tested finding — which is exactly why this chapter is titled around a hypothesis, and why the honest quiz below asks you to separate what the paper measured from what it is proposing to measure next.

Chapter 7: When Humans Disagree, Models Miss It

Chapter 6 hypothesized that reward models and LM judges might be part of what drives collapse, by rewarding “typical” over “different-but-equally-good.” This chapter tests a closely related, more directly measurable question: when real humans genuinely disagree about which of two open-ended responses is better — the exact situation where pluralism, not a single ground truth, is the honest state of affairs — do today's automated evaluators even notice?

This chapter is a genuine change of subject from Chapters 2 through 5, worth flagging explicitly. Every chapter so far measured what models generate. This chapter measures something different: how well the automated tools used to train and evaluate those models can recognize genuine quality when the population of human judges themselves is split. The connection to generation is indirect but load-bearing: whatever these evaluators reward during training is what future generations of models will be pushed toward producing, which is exactly why Chapter 6 needed this chapter's results to make its own hypothesis testable at all.

Building a dataset dense enough to measure disagreement

Most alignment datasets collect a handful of ratings per example — often just 3 — which is nowhere near enough to distinguish “everyone agrees this is a 4/5” from “half of annotators love it, half hate it, and it averages to a 4/5 by coincidence.” The paper fixes this by collecting 25 independent human annotations per item, from Prolific workers paid an average of $15/hour, screened for English fluency and annotation reliability.

Who are these annotators? The paper's demographic table reports 2,296 total unique annotators recruited through Prolific across the study, 95.0% of whom disclosed demographic information. They span 38 different nationalities (the top three: United States 43.5%, United Kingdom 29.3%, Canada 12.5%), average 39.9 years old (standard deviation 13.4, meaning this is not a study skewed entirely toward very young or very old raters), and split close to evenly by sex (male 50.3%, female 49.2%). Education runs the full range the prescreening allowed: undergraduate degree 40.8%, graduate degree 21.4%, high-school diploma 20.4%, technical or community college 13.9%, doctorate 3.5%. Every annotator was screened for English fluency, an approval rate of 99–100%, and 100–10,000 prior Prolific submissions — a genuinely experienced pool, not first-time or low-quality raters. The disagreement measured in this chapter is not a handful of confused participants; it is a broad, demographically varied population of experienced, screened annotators still landing on meaningfully different judgments for the same open-ended response.

With 25 independent judgments per item, the paper can compute Shannon entropy across the label distribution for each item — a standard measure of how spread-out the annotators' opinions were. High entropy means real, substantial disagreement; low entropy means near-consensus. Given the open-ended nature of these queries, the paper finds entropy skews high across a large share of examples — exactly the predictable consequence of asking humans to rank alternatives when several genuinely comparable answers exist.

Three types of automated evaluator, tested against the humans

The paper checks three families of automated quality signal against the 25-annotator human average: LM perplexity (how probable a response is under a language model — lower perplexity treated as a proxy for higher quality), reward model scores (6 top-ranked models from RewardBench, each outputting a scalar preference score), and LM judge ratings (GPT-4o and the open-source Prometheus model, each prompted with either an overall-quality rubric or an HHH — Helpful, Honest, Harmless — rubric derived from the Constitutional AI framework). On the full dataset, none of these three signals correlates strongly with average human judgment to begin with:

Those are modest correlations even before looking at any subset — a Spearman correlation around 0.3 to 0.36 means the automated signal tracks human preference only loosely across the dataset as a whole.

Look at what an LM-judge evaluator actually receives, from the paper's own appendix prompt (Figure 23, lightly condensed). The judge is handed the user's request, the model's response, and a rubric, then asked to return a structured score:

prompt excerpt (paper's Overall Quality LM-judge rubric)
[User Request]: Write a pun about cats.
[Model Response]: I'm feline good about this!
[Evaluation Rubric]:
- Overall Quality: Does the response have overall high quality for
  achieving the user's goal?
[Output]: {"Overall Quality": {"score": 5, "reason": "..."}}

The HHH variant (Figure 24) swaps in three separate criteria — Helpfulness, Harmlessness, Honesty, each scored 1–5 — drawn directly from the Constitutional AI framework's own rubric structure. Notice what this prompt design implies: the judge is asked to produce one number per criterion, per response, looking at that response in isolation. Nothing in the prompt asks the judge to consider “is this meaningfully different from, but equally as good as, some other valid response to the same request?” — the entire evaluation is framed around absolute quality, not relative standing among a plurality of comparably good alternatives. That framing gap, baked into a standard, widely-used judge prompt template, is a plausible mechanical explanation for why LM-judge correlation with human ratings degrades specifically on high-disagreement, high-pluralism examples: the prompt never asks the model to reason about pluralism at all.

What happens to all three evaluator types on exactly the examples where genuine pluralism is most present?

Where the correlation goes, as disagreement concentrates

The paper ranks all (query, response) pairs by annotator disagreement (via Shannon entropy across the 25 labels) and recomputes correlation on progressively smaller, progressively more-disagreed-upon subsets: the top 120, 90, 60, and 30 most disputed examples out of the full pool.

Follow the LM Perplexity column all the way down: 0.361 → 0.170 → 0.045 → −0.030 → −0.108. On the 30 examples where humans disagreed with each other the most, LM perplexity's correlation with average human quality is not just weak — it is negative. On exactly the cases that are the clearest real-world instances of legitimate pluralism (multiple valid, comparably good answers that different people genuinely prefer differently), the cheapest and most widely used automated quality signal in the field points in the wrong direction more often than the right one. Reward model scores follow the identical pattern, turning negative on the most-disputed 30. LM judge scores degrade less catastrophically but still fall from 0.305 to 0.070 — a 77% relative collapse in signal.

Is this robust, or one lucky way of measuring "disagreement"?

“Entropy over 25 labels” is one specific way to quantify disagreement, and a careful reader should ask whether the crash-to-negative result is an artifact of that specific choice. The paper's appendix (Table 20) checks this directly, recomputing the identical top-N analysis under five other disagreement metrics: entropy computed over a 3-way grouping instead of the full 5-point scale (“Entropy Grouped”), Gini impurity (the probability two randomly chosen annotators disagree), raw pairwise disagreement (the fraction of all annotator pairs that differ), Majority vs. Minority (what fraction of annotators did not pick the majority label), and Fleiss' kappa (chance-corrected inter-annotator agreement).

This is a more honest picture than a single clean number would suggest. Three of the six disagreement metrics — Entropy, Entropy Grouped, and Gini Impurity — agree closely with the headline −0.108 result: LM perplexity's correlation genuinely crosses into negative territory on the most-disputed cases under all three. The other three — Pairwise Disagreement, Majority vs. Minority, and Fleiss' Kappa — stay positive on this particular column, though every one of them is still noticeably lower than the 0.361 full-dataset baseline. The finding “correlation degrades sharply as disagreement concentrates” survives being tested six different ways; the stronger claim “the correlation always crosses to negative” depends on which specific disagreement metric you use. Reporting both halves of that picture, rather than only the most dramatic metric, is the more trustworthy way to read this chapter's evidence.

A second, independent test: correlation on "equally good" responses

Disagreement is not the only lens the paper uses. A separate analysis asks a related but distinct question: if you filter down to (Query, Response) pairs that humans rated as roughly equal in quality — using Tukey's fences, a standard outlier-detection technique that discards ratings outside Q1 − k·IQR to Q3 + k·IQR — do the automated evaluators still track human judgment? The paper computes this two ways: filtering “locally” within one query's own response pool (L), and filtering “globally” across the entire pool of responses to every query (G).

These two rows tell genuinely different stories, and both are worth knowing. Filtered locally — comparing a response only against other responses to the same query — perplexity's correlation with human quality barely moves, and even rises slightly at the 60% subset (0.412). Filtered globally — comparing across the entire pool, so “similar quality” can mean similarly-rated responses to completely different prompts — correlation drops by more than half at the tightest 40% subset, from 0.361 down to 0.149. Reward model and LM judge scores follow the same global-versus-local pattern: at the 40% global subset, reward model correlation falls to 0.096 and LM judge correlation to 0.121, both roughly a third of their full-dataset values. The takeaway is more precise than “automated evaluators fail on similar-quality responses” in general: they specifically struggle to distinguish quality among responses to different prompts that happen to share a similar quality tier — a genuinely harder calibration problem than ranking alternatives to one fixed query, and one production reward models are routinely asked to solve anyway, since they must score responses across an entire deployment's worth of different user prompts.

Closing the loop back to Chapter 6

This is the empirical half of the typicality-bias mechanism. If reward models and LM judges are least reliable exactly on the cases where real humans hold genuinely divided, idiosyncratic preferences — and most reliable on cases with clear consensus — then any training pipeline that optimizes a model against these signals is, by construction, learning to avoid exactly the response space where legitimate diversity of good answers lives, because that is precisely the region its own evaluator cannot score consistently. The evaluators are not merely imperfect at measuring diversity; on the highest-diversity-of-opinion cases, they actively mislead.

Trace the causal arrow all the way through, chapter by chapter, and it forms a closed loop. Chapter 6 proposed that RLHF's reward-model optimization is mode-seeking because it cannot reliably tell “objectively worse” from “just less typical.” This chapter supplies the direct empirical half of that claim: reward models and LM judges genuinely do lose their signal exactly on the cases where the two are hardest to distinguish — the high-disagreement, high-pluralism examples. Put together, the loop reads: models are trained against evaluators (Ch 6) → those evaluators are least reliable on exactly the response space where legitimate diversity lives (Ch 7) → training against an unreliable-there signal squeezes probability away from that space regardless of the evaluator's intent (Ch 6, the KL asymmetry) → the resulting models converge on the “safe,” evaluator-favored regions (Ch 2–4) → and the whole cycle repeats whenever those models' outputs become the next round's training or evaluation data (Ch 8). No single chapter proves the loop causally end to end — that is exactly what Chapter 6 flagged as unproven — but every link in the chain has direct measured support somewhere in this session.

Chapter 8: Showcase: The Synthetic Data Spiral — what this means for training the next model

Every prior chapter measured collapse in models people already use. This chapter follows the consequence forward: what happens when today's homogenized model outputs become tomorrow's training data?

This is the “showcase” chapter of the session for a specific reason: it is the one place where every prior chapter's finding gets combined into a single forward-looking argument. Chapter 2's intra-model collapse means one teacher's outputs are already narrow. Chapter 4's inter-model homogeneity means a swarm of teachers does not rescue that narrowness the way it looks like it should. Chapter 6's typicality-bias mechanism supplies the reason the narrowing happens in the first place. This chapter asks what all three facts, taken together, predict about a training pipeline that increasingly runs on model-generated rather than human-generated data — and, unlike the earlier chapters, has to build that prediction itself, honestly labeled as extrapolation, because the paper does not run this exact multi-generation experiment.

Why model-generated training data is now unavoidable

Distilling a smaller or cheaper model by training it on a larger model's outputs is one of the most common, genuinely useful techniques in the field — it is fast, it is far cheaper than fresh human-annotated data, and it works. But the paper's Broader Implications section is direct about the known risk this carries: relying on a single model as the teacher for synthetic data generation can intensify mode collapse, reinforcing whatever narrow response patterns that one teacher already favors, and in the worst case producing degenerative feedback loops — the paper cites prior work by Alemohammad and colleagues, memorably titled “Self-consuming generative models go MAD,” and by Dohmatob, Feng, Subramonian, and Kempe on “Strong model collapse,” both documenting this dynamic mathematically: a generative model trained repeatedly on its own (or a closely related model's) outputs systematically loses fidelity to the original, real-world data distribution across successive generations.

Connect this to a term this session already gave a precise meaning: “losing fidelity to the original distribution” is, in Chapter 6's vocabulary, exactly what a mode-seeking process does to a mode-covering one. Generation 1's collapsed teacher already lost some of the real distribution's spread when it was trained (Chapter 6's typicality-bias mechanism). Generation 2, trained on generation 1's already-narrowed outputs, is learning to imitate a target distribution that was never the real, human-authored one to begin with — it is imitating an imitation. Each successive generation inherits not the original diversity, but whatever fraction of it survived the previous round's collapse, which is precisely why the decay compounds geometrically rather than resetting back toward the true distribution every time a new model is trained from scratch on the corrupted corpus.

The proposed fix, and why this paper's own data undermines it

The field's standard answer to single-teacher collapse is to diversify the teacher: use model swarms or multi-agent frameworks that pool outputs from several distinct models, on the reasonable-sounding theory that different companies' models will bring different perspectives, the way a panel of different human experts would.

Chapter 4's numbers are the direct rebuttal to that theory. If DeepSeek-V3 and Qwen-Max average 0.82 similarity to each other on open-ended queries, and DeepSeek-V3 and GPT-4o average 0.81, then an “ensemble” built from those three models is not three independent sources of diversity — it is something closer to one and a fraction. The paper states this conclusion explicitly: “even across distinct state-of-the-art models, diversity in open-ended tasks is far from guaranteed. Models often converge toward highly similar answers, undermining the assumed benefits of multi-model distillation.” Practitioners building a “diverse” synthetic-data pipeline out of several frontier APIs may be paying for three API bills and getting something much closer to one API's worth of actual diversity.

How much does adding a third or fourth teacher model actually buy you?

Push the swarm argument one step further than the paper does, with an honest, hand-derived toy model (this lesson's own extension, not a paper-reported result). Suppose you assemble N teacher models for a synthetic-data pipeline, and every pair of them shares the same average pairwise similarity s, drawn from Chapter 4's real observed range (0.71–0.82). Model 1 contributes one full “unit” of coverage. Model 2, being s similar to Model 1, contributes only the non-overlapping fraction, (1−s) units of genuinely new coverage. If Model 3 is similarly redundant with the combined pool at the same rate s, it adds only (1−s) of what remains uncovered — (1−s)² units — and so on. Total coverage after N teacher models is a geometric series:

python
def swarm_coverage(s, N):
    r = 1 - s
    return sum(r**k for k in range(N))

for s in [0.82, 0.71]:  # the paper's real observed inter-model similarity range
    for N in [1, 2, 3, 10]:
        print(s, N, round(swarm_coverage(s, N), 4))
# s=0.82:  N=1 -> 1.00   N=2 -> 1.18   N=3 -> 1.21   N=10 -> 1.2195 (the asymptote, 1/s)
# s=0.71:  N=1 -> 1.00   N=2 -> 1.29   N=3 -> 1.37   N=10 -> 1.4084 (the asymptote, 1/s)

As N grows without bound, that geometric series converges to 1 ÷ s — a hard ceiling. At the tighter end of the paper's measured similarity range (s = 0.82, roughly DeepSeek-V3-and-Qwen-Max territory), no swarm of arbitrarily many such models — 3, 10, 100 — ever exceeds about 1.22× the coverage of a single teacher model. At the looser end (s = 0.71), the ceiling is a somewhat better but still modest 1.41×. Compare that to what a naive practitioner might assume: three teacher models “should” give roughly 3× the coverage of one. The real ceiling, using this paper's own measured redundancy, is nowhere close — and it is a hard mathematical ceiling that adding a fourth, fifth, or fiftieth similarly-redundant model cannot break.

The lever this toy model exposes is not “how many models,” it is s itself — the pairwise similarity between teacher models. A practitioner who wants to raise the coverage ceiling has exactly one number worth spending engineering effort lowering: the actual measured redundancy between candidate teacher models, on their specific task distribution, not the count of distinct vendor names on the invoice. Two models at s = 0.5 (a genuinely more independent pair than anything Chapter 4 measured among today's frontier models) would cap out around 1 ÷ 0.5 = 2× a single model's coverage — still not infinite, but a meaningfully better ceiling than the 1.22× this chapter's real measured numbers produce. Finding or training genuinely lower-s teacher models, not simply adding more of the same kind, is where this math says the actual leverage lives.

Modeling the compounding, honestly

The paper does not run a multi-generation distillation experiment measuring diversity decay directly — that extension is exactly the kind of thing its own future-work section gestures at. What follows is this lesson's own illustrative extrapolation, built from the paper's real, measured inter-model similarity range (71%–82%), to make the compounding mechanism concrete. Treat every number past this sentence as a toy model, not a paper-reported result.

Define retention as the fraction of a corpus's diversity that is genuinely new information, rather than redundant with what a single generation's dominant teacher-model output already covers — a reasonable proxy is retention = 1 − average pairwise similarity, using the paper's own 71%–82% observed range, which puts retention somewhere between 0.18 and 0.29 per generation of distillation. If diversity decays geometrically at that retention rate — the simplest possible model of repeated, compounding information loss — then after n generations of a model trained on the previous generation's synthetic output:

python
def diversity_after_generations(retention, n_generations, d0=1.0):
    return d0 * (retention ** n_generations)

# using the paper's real 71%-82% inter-model similarity range as the retention parameter
for retention in [0.18, 0.29]:  # 1 - 0.82, 1 - 0.71
    for n in [1, 2, 4]:
        print(retention, n, round(diversity_after_generations(retention, n), 4))
# 0.18 1 0.18   0.18 2 0.0324   0.18 4 0.00105  -- a single collapse-prone teacher
# 0.29 1 0.29   0.29 2 0.0841   0.29 4 0.00707  -- a more diverse teacher, still decaying fast

At either end of the paper's real observed similarity range, this toy extrapolation loses the overwhelming majority of the corpus's original diversity within just two to four rounds of self-distillation, purely from compounding. Whether or not the exact exponent is right, the qualitative shape — fast, compounding, geometric decay, not slow linear drift — matches exactly what the “self-consuming generative models go MAD” and “strong model collapse” papers describe.

Combining both toy models: does a swarm actually rescue the compounding?

The two derivations built so far in this chapter — the swarm-coverage ceiling above, and the generation-over-generation decay just shown — are usually discussed separately. Put them together (still entirely this lesson's own illustrative extension, not a paper-reported result) and ask the practical question a team actually building a synthetic-data pipeline would ask: does switching from a single teacher to a 3-model swarm meaningfully slow the compounding collapse?

Take the single-teacher retention rate at the tighter end of the paper's range (retention = 0.18, from s = 0.82) and scale it up by the swarm-coverage multiplier this chapter already derived for N = 3 models at that same similarity (1.2124×, from the geometric-series formula above), giving a swarm-adjusted retention of roughly 0.18 × 1.2124 ≈ 0.218:

python
single_teacher_retention = 0.18
swarm_multiplier_N3 = 1.2124   # from swarm_coverage(s=0.82, N=3), derived earlier this chapter
swarm_retention = single_teacher_retention * swarm_multiplier_N3

for label, r in [('single teacher', single_teacher_retention), ('3-model swarm', swarm_retention)]:
    for n in [1, 2, 4]:
        print(label, n, round(r ** n, 5))
# single teacher:  n=1 0.180    n=2 0.032    n=4 0.00105
# 3-model swarm:   n=1 0.218    n=2 0.048    n=4 0.00227

By generation 4, the swarm-fed pipeline retains about 0.227% of its original diversity versus the single-teacher pipeline's 0.105% — roughly double, which sounds like meaningful relief until you notice both numbers round to “essentially nothing.” A 3-model swarm delays the collapse by a fraction of one generation's worth of decay; it does not change the qualitative outcome. This is the sharpest, most quantitative version of this chapter's core warning: model swarms are a real, measurable improvement over a single teacher, exactly as this chapter's earlier coverage-ceiling math showed — and that improvement is nowhere near large enough to rescue a synthetic-data pipeline from compounding collapse across multiple generations of self-distillation, because the swarm's redundancy problem and the compounding problem are multiplying against each other, not offsetting each other.

The direct bridge to this site's own curation lessons

This is exactly why the site's semdedup-datacomp.html veanor on curation-by-embedding matters alongside this one. SemDeDup and similar embedding-based deduplication techniques remove near-duplicate examples from a training corpus by exactly the mechanism Chapter 2 of this lesson used to diagnose collapse — embed everything, compute pairwise similarity, prune what's redundant. That is a genuinely valuable technique for cleaning noisy web-scraped data. But it has a blind spot this chapter's math exposes: if the corpus being deduplicated was already generated by one or a few homogenized models, then heavy deduplication is pruning duplicates of duplicates — it makes the corpus smaller and cleaner, but it cannot manufacture diversity that was never in the source distribution to begin with. Dedup finds and removes what's redundant. It does not invent what's missing.

Make the ceiling concrete with a small toy count (illustrative arithmetic, not a paper-reported number). Suppose a synthetic corpus of 10,000 examples was generated by asking one collapsed teacher model for “a metaphor about time,” and — consistent with Chapter 0's calibration — the teacher effectively draws from only k ≈ 1.27 genuinely distinct ideas. Perfect, maximally aggressive semantic deduplication would collapse those 10,000 examples down to somewhere near k canonical representatives — call it 2, one per surviving cluster. That is exactly what SemDeDup is designed to do, and it is a real, valuable cleanup: 10,000 near-duplicate rows shrink to 2 information-dense ones, saving enormous training compute for equivalent signal. But no dedup threshold, however aggressive, turns those 2 canonical ideas back into the 10 or more a genuinely diverse human population would have produced. The corpus gets smaller. It does not get wider. Fixing “wider” requires fixing generation — better decoding (Chapter 3, real but partial), a broader teacher roster with genuinely low pairwise similarity rather than an ensemble that just looks broad (Chapter 4, this chapter), or a training-level fix at the alignment stage itself (Chapter 6, currently a hypothesis) — not a smarter filter applied after the fact.

Chapter 9: Beyond & Connections

This session has stacked up nine chapters of evidence that mode collapse is real, structural, and survives every naive fix a user or a decoding engineer could reach for. It closes by being precise about what has actually been tried and verified, versus what remains proposed and untested — the same honesty the paper itself insists on in its own Limitations section.

Read the nine chapters as a single argument rather than nine separate facts and the shape becomes clear: Chapter 0 established the phenomenon is real and not an artifact of prompt selection. Chapters 2 through 5 measured it from every angle a skeptic could raise — within one model, across models, under aggressive decoding, under paraphrasing — and it survived every single check. Chapter 6 proposed a mechanism. Chapter 7 supplied that mechanism's empirical half. Chapter 8 followed the consequences forward into the training pipelines being built right now. Nine chapters, one throughline: the narrowness is real, it is structural, and it is not something a user, a prompt, or a decoding parameter can fix from outside the model.

Reproducibility: this is a released benchmark, not just a set of numbers to trust

Every measurement in this session is independently checkable, which matters for a paper whose central claim is this uncomfortable. The authors published both the code (github.com/liweijiang/artificial-hivemind) and the Infinity-Chat dataset itself as a public HuggingFace collection (liweijiang/artificial-hivemind). That means the entire measurement pipeline this session walked through — the filtering funnel from Chapter 1, the embed-and-cosine-similarity protocol from Chapter 2, the min-p comparison from Chapter 3 — is not something you have to take on faith from a PDF. If you have API access to a handful of models and an embedding endpoint, you could rerun a scaled-down version of Chapter 2's core experiment yourself on the released 26,070-query benchmark, and check whether your own numbers land anywhere near this session's headline 79%-above-0.8 figure. A diversity claim this counterintuitive earns much more trust from being reproducible than from being merely well-argued.

Closing the loop on Chapter 0's second claim

Chapter 0 flagged that the paper's abstract packs in two separate claims — that models struggle to be diverse, and that repeated exposure to their narrow output could homogenize human thought over time — and promised to return to the second one here. Be precise about what this session can and cannot tell you about it. Every chapter of hard evidence in this lesson measures the first claim: model outputs are narrow, in ways that survive resampling, rewording, and model-swapping. The second claim, about downstream effects on human cognition and culture, is not something Infinity-Chat measures directly at all — it is the motivating concern the paper opens with, not a result it reports. The paper's own societal-implications section is careful on this exact point, describing “emerging evidence” of shifts in human writing and ideation following widespread chatbot adoption as evidence from other cited studies, not a finding this paper's own experiments produced. If you came away from this session believing “this paper proved AI is making people less creative,” that overstates what was actually measured. What was actually measured, exhaustively, is the upstream half of that concern: the models themselves are narrow. Whether that narrowness is already reshaping human thought at scale is a real, serious, and still largely open empirical question — exactly the kind of question a rigorous measurement tool like Infinity-Chat exists to eventually help answer, not one it has answered yet.

What this paper actually tested, and what it only cites

The paper's own honest limitations, in full

Five worth carrying forward, in the paper's own words where it matters most. First, “Infinity-Chat represents only a snapshot of the vast space of possible open-ended queries and may not capture all forms of creative divergence across different contexts” — 26,070 queries, however carefully filtered and validated, is still a sample, not the whole space of things people might ask a chatbot. Second, Infinity-Chat is English-only, mined from WildChat's user base — the authors expect similar homogenization to hold across languages and cultures given how globally overlapping pretraining data and alignment practices already are, but say so explicitly as an expectation, not a demonstrated result, and commit to strengthening that discussion in a future camera-ready revision.

Third, the taxonomy classifier (gpt-4o-2024-11-20) agrees with human annotators 74.7% of the time — comparable to typical human-to-human annotation agreement, but not perfect, so some category boundaries in Chapter 1's table carry real uncertainty. Fourth — and this is the limitation Chapter 2 already leaned on directly — “relying on semantic similarity of text embeddings to quantify diversity may lack sufficient expressiveness to capture the full spectrum of creative variation.” A cosine-similarity score is a proxy, not a perfect oracle for “these two ideas are meaningfully different.” Fifth, and most fundamentally: everything in this session is correlational. The paper documents that collapse happens, in exhaustive, carefully-controlled detail — it does not, and says it does not, establish which specific factor among pretraining data, alignment, memorization, or generalization is causally responsible. That is precisely the gap Chapter 6's typicality hypothesis fills speculatively, and precisely what the paper's own future-work section is designed to close.

Read those five limitations again as a single group and a pattern emerges: every one of them is the paper being honest about the boundary of its own method, not a flaw discovered by an outside critic. A paper willing to list “our own diversity metric might not be expressive enough” as its own fourth limitation, in its own words, is behaving exactly the way this session has tried to model throughout — distinguishing “measured” from “inferred” from “suspected,” and saying so plainly at every step rather than letting a strong headline number imply more certainty than the underlying method can support.

How this study positions itself against prior mitigation work

The paper's related-work section is candid that diversity collapse is not a problem it discovered — prior work already identified contributing factors including training on synthetic data, the alignment process itself, and insufficient diversity in training data to begin with, and already proposed mitigations spanning training-corpus diversification, training-algorithm modifications, alternative decoding strategies, and alternative prompting strategies. What Infinity-Chat contributes is not a new mitigation technique to add to that list — it is the first large-scale, real-world measurement instrument rigorous enough to tell which of those proposed mitigations (like min-p, tested here in Chapter 3) actually move the needle, and by how much, on realistic open-ended traffic rather than synthetic proxy tasks. That distinction — measurement infrastructure versus a new fix — is worth holding onto: this paper's contribution is a ruler, not a repair.

If you want to go deeper: three research threads the paper connects to

The related-work section situates Infinity-Chat against three separate literatures, each a legitimate next stop for a reader who wants to go past this one lesson.

Measuring creativity psychometrically. A separate line of work adapts tests originally designed to measure human creativity onto language models: the Divergent Association Task (asking a model to generate words that are semantically as unrelated as possible from each other, then scoring how far apart they land), the Alternate Uses Test (asking for unusual uses of an everyday object, a classic psychometric probe of divergent thinking), and the Torrance Tests of Creative Thinking (a decades-old standardized battery originally built for human students, adapted here for models). These tests predate Infinity-Chat and measure something related but narrower: an individual model's raw creative-generation ability on a fixed, small task, rather than population-scale convergence across many models on realistic, open-ended chatbot traffic.

Pluralistic alignment. A growing subfield argues that AI alignment has historically assumed a single, monolithic notion of “the right answer,” and pushes instead for training and evaluation methods that explicitly represent a population's range of legitimate preferences rather than averaging them away. Chapter 7's 25-annotator, high-entropy dataset is this session's direct contribution to that literature: dense enough distributional data to actually detect when consensus is fake and disagreement is real, rather than silently averaging a divided population into one misleading number.

Algorithmic monoculture. A third, more societal-scale thread asks what happens when many independent decision-makers — hiring pipelines, content recommenders, now chatbots — all rely on similar underlying models, and whether that produces correlated, homogenized outcomes across a whole population even though each individual model looks reasonable in isolation. Chapter 4's inter-model homogeneity finding, and Chapter 8's swarm-coverage ceiling, are this session's concrete numbers for exactly that concern, applied specifically to open-ended generation rather than classification or ranking.

Notice that these three threads sit at three different scales of the same underlying worry, from smallest to largest. Creativity psychometrics asks about one model, one task: how much genuine range does a single system show when probed directly. Pluralistic alignment asks about one model, one population of users: whether that system's training process even has the capacity to represent a population's real spread of preferences rather than flattening it to one consensus. Algorithmic monoculture asks about many systems, one population: whether a whole ecosystem of nominally-competing AI products ends up delivering the same narrow set of outcomes to everyone who uses any of them. This session's own two central findings map onto the second and third of those scales directly — intra-model repetition (Chapters 2–3) is a pluralistic-alignment-scale problem, one model failing one population; inter-model homogeneity (Chapter 4, Chapter 8) is an algorithmic-monoculture-scale problem, many models failing the same way at once. The first scale, individual model creativity, is the one prior psychometric work already covers reasonably well — which is exactly why Infinity-Chat's real contribution sits at the two larger, less-studied scales.

Comparison: how this study differs from what came before it

A practitioner's checklist, built entirely from this session's own numbers

Turn nine chapters of measurement into something actionable, without adding a single claim beyond what this session already established.

Where this connects across the rest of this course

This session sits at a junction of three earlier ones. vector-embeddings.html and similarity-metrics.html supplied the exact measurement tool — cosine similarity over sentence embeddings — that every chapter here relied on. Session 06's cs8803llm-06-dpo-alignment.html derived the KL-divergence “leash” that Chapter 6 leaned on to explain mode-seeking behavior in preference optimization. And semdedup-datacomp.html shows the mirror-image technique — embedding-based curation used to clean a corpus — which Chapter 8 argued cannot manufacture diversity a collapsed source distribution never had.

One more connection worth naming explicitly: this course's session on prompt engineering teaches how to phrase requests to get better, more targeted outputs from a model. Chapter 5 of this session is the direct limit case of that skill — a demonstration that no amount of careful rewording reaches past a certain point, because the thing being asked to change (which family of ideas a model retrieves) lives upstream of anything a prompt can touch. Treat the two sessions as complementary, not contradictory: prompt engineering is genuinely useful for steering a model within the modes it already has access to; it is not a tool for manufacturing modes a model's training never gave it.

Putting this session's own derived numbers side by side

Three numbers this lesson built along the way, none of them lifted directly from the paper's own tables, turn out to be telling the same story from three different angles. Chapter 0's back-of-envelope calibration found that a 79%-of-cases-above-0.8 collision rate implies real models behave as if drawing from roughly k ≈ 1.27 effective ideas, out of a space that plausibly contains ten or more. Chapter 5's hypothesis-scale placement found the paraphrase-robustness gap sits 94% of the way toward pure concept-level retrieval and only 6% toward string memorization. Chapter 8's swarm-coverage derivation found that even an unlimited number of 82%-similar teacher models caps out at roughly 1.22× the coverage of a single model. Three independent toy calculations, built from three different real measurements in three different chapters, all land on the same qualitative conclusion: whatever “the space of good answers” genuinely contains, a model trained the way today's models are trained explores only a small, tightly bounded slice of it, and no amount of resampling, rewording, or model-swarming reaches meaningfully past that slice.

Herbert Simon's observation about attention applies just as well to the answers these systems hand back: scarcity is the whole story. There is no scarcity of good metaphors for time. There is a scarcity of what a model, trained the way today's models are trained, is willing to show you. As the paper's own societal-implications discussion puts it, “as billions of users increasingly depend on LLMs for creative, educational, and decision-making purposes, understanding and quantifying behavioral homogenization becomes critical” — which is exactly the measurement problem Infinity-Chat exists to solve, and exactly why this session insisted, chapter after chapter, on separating what has been measured from what is still merely suspected.

References

1. Jiang, L., Chai, Y., Li, M., Liu, M., Fok, R., Dziri, N., Tsvetkov, Y., Sap, M., Albalak, A., Choi, Y. “Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond).” 2025. arXiv:2510.22954
2. Nguyen, M., Baker, A., Neo, C., Roush, A., Kirsch, A., Shwartz-Ziv, R. “Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputs.” ICLR 2025. (min-p decoding, tested directly in Chapter 3)
3. Alemohammad, S., Casco-Rodriguez, J., Luzi, L., Humayun, A. I., Babaei, H., LeJeune, D., Siahkoohi, A., Baraniuk, R. “Self-Consuming Generative Models Go MAD.” ICLR 2024. (cited by Hivemind for the synthetic-data feedback-loop risk, Chapter 8)
4. Dohmatob, E., Feng, Y., Subramonian, A., Kempe, J. “Strong Model Collapse.” 2024. arXiv:2410.04840 (cited by Hivemind for the synthetic-data feedback-loop risk, Chapter 8)
5. West, P., Potts, C. “Base Models Beat Aligned Models at Randomness and Creativity.” 2025. (cited by Hivemind as related evidence for the alignment/diversity trade-off, Chapter 6)
6. Bai, Y. et al. “Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback.” 2022. (source of the HHH rubric used by the LM-judge evaluation, Chapter 7)
7. Bommasani, R., Creel, K. A., Kumar, A., Jurafsky, D., Liang, P. S. “Picking on the Same Person: Does Algorithmic Monoculture Lead to Outcome Homogenization?” NeurIPS 2022. (the algorithmic-monoculture literature this session's inter-model findings connect to, Chapter 9)
8. Zhao, W. et al. “WildChat: 1M ChatGPT Interaction Logs in the Wild.” 2024. (source corpus that Infinity-Chat's 37,426 candidate queries are mined and filtered from, Chapter 1)