Curation by Embedding

Chapter 1: Why Scale Alone Fails

The scaling-law era taught a simple reflex: if the model is not good enough, add data. That reflex is correct in a specific, narrow setting — when the added data is independently drawn from the distribution you care about. Web crawls violate that condition in three separate ways, and each violation has a mechanism you can compute.

Let us take them one at a time, because the fixes are different and people constantly conflate them.

Failure 1: duplication is silent importance weighting

Start with what a training run actually does. In one epoch of stochastic gradient descent over a corpus of N examples, the total parameter update is the sum of per-example gradients:

Now suppose one particular example — a stock photo of a golden retriever on a lawn, say — appears m times in the corpus, because it was scraped from m different sites that all licensed it. Its gradient enters the sum m times. The optimiser cannot tell the difference between "this example appears 50 times" and "this example appears once and I weighted it 50×."

Make it concrete. A corpus of 1,000 examples where one image appears 50 times:

Five percent of your model's learning, forever, devoted to one picture. Ask yourself whether you would ever choose that weighting. Nobody would. But nobody chose it — it arrived through a scraper, and it is invisible unless you go looking.

The measurable downstream harms are well documented. Duplicated training text is memorised and regurgitated far more readily than unique text — the deduplication work on language corpora found that removing duplicates cut the rate at which models emit memorised training strings by an order of magnitude, and improved held-out perplexity at the same time. Duplicates also smuggle test-set contamination past your splits: if the same image appears in the crawl under two URLs and one lands in your evaluation set, your evaluation is now partly a training-set lookup.

Three kinds of duplicate, and why only one is easy

That third row is the whole of SemDeDup's contribution. Its central empirical claim — that roughly half of a large LAION subset is semantically redundant — is a claim about a category of duplicate that byte-level and pixel-level tools are structurally blind to.

Failure 2: a mismatched pair is not neutral, it is an instruction

The second failure mode is noise, and contrastive training has an unusually vicious relationship with it. Consider what a web caption actually is. Alt-text was invented for screen readers and is, in practice, filled with whatever the site's CMS put there:

Only the fourth is a description. The first is a filename. The second is navigation. The third is search-engine keyword soup. In a supervised classifier, a bad label corrupts exactly one example: the model gets one wrong instruction and moves on.

In contrastive training, the pairing is the label, and the loss compares every item in the batch against every other. So one bad pair does not damage one relationship. Work it out. With a batch of N pairs, the contrastive objective asserts:

• image i should be closest to caption i — 1 assertion
• image i should be far from all N−1 other captions — N−1 assertions
• caption i should be far from all N−1 other images — N−1 assertions

At a batch size of 1,024 — unremarkable for CLIP training — a single mismatched pair issues 2,047 wrong instructions. It pulls an image toward a caption that does not describe it, pushes that caption away from 1,023 images including ones it might genuinely describe, and pushes that image away from 1,023 captions including its true one if it happens to be in the batch.

Failure 3: the web is Zipf-shaped, and scale multiplies the head

The third failure is the least discussed and, at the frontier, probably the most limiting. Concept frequencies on the internet follow a heavy-tailed law. Approximate it with Zipf: the r-th most common concept has frequency proportional to 1/r.

What fraction of the data do the top 100 concepts get, out of a million concepts? The total mass is the harmonic number, which for large n is well approximated by

So the top 100 out of a million:

One hundredth of one percent of the concepts take 36% of the data. Now do the thought experiment that matters: you double your crawl. Every concept's count doubles. The proportions do not move. You have twice as many pictures of dogs and cars, and you have gone from four photos of a Namaqua chameleon to eight.

The three failures, side by side

Read the second column and the third column together. In every row, the raw-bytes view offers no purchase and the embedding view offers a number you can threshold. That is the entire reason this lesson exists: the embedding turns three qualitative complaints about data into three quantities.

A worked audit you could run on any corpus tonight

Before any algorithm, here is the diagnostic. Given a corpus, embed a random sample of 100,000 items and compute three statistics.

python — the three-number data auditimport numpy as np

E = embed(sample)                        # (100_000, d), then L2-normalise
E = E / np.linalg.norm(E, axis=1, keepdims=True)

# 1. REDUNDANCY: what fraction has a very close neighbour?
#    (exact for 100k; use FAISS for more)
S = E @ E.T
np.fill_diagonal(S, -1)
nn_sim = S.max(axis=1)                    # nearest-neighbour cosine per item
print("frac with NN cosine > 0.95:", (nn_sim > 0.95).mean())

# 2. ALIGNMENT (paired data only): image-text cosine distribution
align = (E_img * E_txt).sum(axis=1)
print("median align:", np.median(align), " frac < 0.20:", (align < 0.20).mean())

# 3. SKEW: cluster occupancy entropy, in bits
labels = kmeans(E, k=1024).labels_
p = np.bincount(labels, minlength=1024) / len(labels)
p = p[p > 0]
H = -(p * np.log2(p)).sum()
print("entropy:", H, "of max", np.log2(1024), " effective clusters:", 2**H)

Three numbers: near-duplicate fraction, alignment fraction, and effective cluster count. Every method in this lesson is an intervention on one of them. If you take nothing else from this chapter, take the habit of computing them before you train.

What this chapter did not claim

Be careful about the boundary. None of the above says scale is bad or that small clean datasets beat large ones. The scaling laws are real and adding data does help. The claim is narrower and stranger:

That result — that data pruning can bend the scaling curve rather than merely translate it — came out of the same research group that then built SemDeDup, and it is the theoretical licence for the whole enterprise. If pruning could only ever cost you performance, the best you could hope for from curation would be efficiency. Because pruning can improve performance, curation is a source of capability.

Failure 4: your test set is in your training set

There is a fourth failure that is really a consequence of the first, and it deserves its own name because it corrupts your ability to detect the other three.

Benchmark images and benchmark text are, without exception, republished. ImageNet photographs appear in blog posts about ImageNet. Evaluation questions appear in tutorials, in forum answers, in scraped repositories. A web crawl large enough to be useful is large enough to contain your evaluation.

Estimate the exposure. Suppose an evaluation set has 50,000 items, and suppose each has an independent 2% chance of appearing somewhere in a multi-billion-item crawl in a near-identical form. The expected number of leaked items is:

Two percent sounds tolerable until you remember what it does to a comparison. If two models differ by 1.5 accuracy points and one of them was trained on a corpus with slightly more leakage, the entire measured difference can be leakage. And crucially, leakage does not distribute uniformly: it concentrates on the famous examples, which are also the ones every model finds easiest, so it inflates exactly the region where the metric is most saturated.

The gradient budget ledger

The three failures compose, and it is worth doing the arithmetic once, because the compounded number is larger than people expect.

Take a 100-million-pair pool and three independent-ish estimates: 30% of pairs are semantic near-duplicates of something else in the pool, 25% of pairs are misaligned, and the useful remainder is Zipf-skewed so that a large share of it is redundant information even when the rows are distinct.

Two-thirds of the corpus is misaligned, redundant, or piled onto concepts you already had in abundance. That number is a rough estimate built from rough inputs, and it is close enough to what the measured results in later chapters imply that it should recalibrate your intuition about what "a hundred million pairs" means.

Memorisation, and the mechanism behind it

Duplication has one more consequence that is not about efficiency at all.

A model trained with a next-token or contrastive objective reduces loss on an example either by generalising — learning a rule that covers it and many others — or by memorising it. Generalisation is cheaper in parameters and pays off across examples; memorisation is expensive per example but always available.

Now consider what duplication does to that tradeoff. If an example appears once, memorising it buys a loss reduction on one item. If it appears fifty times, memorising it buys fifty times the loss reduction for the same parameter cost. Duplication makes memorisation a better deal, in exactly the arithmetic sense.

This is why deduplication reduces verbatim regurgitation so sharply, and it is why the effect is nonlinear: an example seen twice is not twice as likely to be memorised as one seen once, because the threshold at which memorising beats generalising is crossed somewhere in between.

So deduplication is simultaneously a compute optimisation, a distributional correction, and a privacy and licensing mitigation. Three arguments, one operation, and the first is usually the only one anyone mentions.

Why nobody fixed this sooner

A fair question, given that none of the above is technically hard. Three structural reasons, and each one is a lesson about how fields allocate attention.

Nobody owned it. Data collection was infrastructure work, done once, credited to nobody, and then treated as settled. Model work was research, published, and cited. The incentive gradient pointed away from the higher-leverage axis for a decade.

It was unmeasurable. Chapter 4's whole argument. Without a controlled comparison, "our filter is better" was an unfalsifiable claim, so filters could not accumulate improvements the way architectures did.

The tool did not exist at the right price. Semantic deduplication requires an embedder good enough to place meaning consistently, cheap enough to run over a billion items. Before large pretrained encoders, the first condition failed. Before efficient inference and vector indices, the second did. The method is obvious in hindsight precisely because its two prerequisites arrived quietly, from elsewhere.

Which failure dominates in your corpus?

The three failures are not equally present everywhere, and the fix for one does nothing for the others. Diagnose before you treat.

Two cautions on reading these signals. First, the nearest-neighbour spike is only visible after exact and perceptual duplicates are removed — otherwise it is swamped by trivial copies and tells you nothing new. Second, the alignment tail is scorer-dependent: a low-scoring pair can be a genuinely bad pair or a perfectly good pair phrased in a way your scorer was never trained on. Read samples from the tail before you trust the shape.

How sensitive is the ledger?

The two-thirds estimate above came from three guessed inputs, so it is worth asking how much the conclusion depends on the guesses. Vary them:

Across a wide, plausible range of inputs the answer moves between roughly a quarter and roughly a half. The conclusion — that most of a raw web corpus is not carrying its weight — is robust to being badly wrong about any single input, which is the mark of an estimate worth making even when you cannot measure the pieces precisely.

Track the three numbers over time, not once

A one-time audit tells you the state of a corpus. Running it on every snapshot tells you about your pipeline, which is more useful.

The general point: these are not one-off research metrics, they are production telemetry. A corpus is a system with inputs and drift, and the same instincts that make you graph latency should make you graph diversity.

The same three failures in text, at different proportions

Everything above was argued with images because that is where the controlled measurements exist. Text has the identical three failures with different weights, and knowing the differences prevents a lot of wasted effort.

The practical consequence is that text pipelines look different in a specific way: exact and near-exact deduplication is the workhorse and does most of the value, quality classification replaces cross-modal scoring, and semantic deduplication is a smaller final pass rather than the headline act.

Worked example: how syndication inflates a text corpus

Take one wire-service article. Estimate its footprint in a crawl:

Now apply the importance-weight argument. Without deduplication, that one story contributes 141 times the gradient of a story published once. The events that get syndicated are, by definition, the ones already over-represented, so the effect compounds skew rather than distributing randomly. And notice how the instruments partition: exact hashing catches 40, shingles add 25 and some of the aggregators, and only a semantic pass touches the last 15. The three stages are not redundant with one another.

Where generated text changes the arithmetic

One newer wrinkle worth naming, because it changes the failure modes rather than merely their magnitudes.

A growing share of web text is machine-generated. For curation this creates a category that fits none of the three failures cleanly: it is not duplicated (each output is distinct), it is not obviously misaligned (it is fluent and on-topic), and it does not create skew in the usual sense (it can be produced about anything). It is fluent, plausible, and low in information — and every quality signal built to detect the old failure modes rates it highly.

The honest state of this problem is that it is open. It is included here not because there is a fix to teach but because it is the clearest current example of the general principle: a curation signal detects the failure modes it was designed against, and a corpus will drift toward whatever the signal cannot see. That principle is the entire subject of Chapter 7.

Chapter 2: SemDeDup, Derived

We want an algorithm that finds, among N examples, the groups that are "the same thing," and keeps one member of each group. Nothing about that sentence mentions clustering, thresholds, or centroids. Those all fall out of two constraints. Let us derive them rather than recite them.

Step 1: define "the same thing" with a number

Take a pretrained encoder f that maps an example to a vector. For LAION images, SemDeDup used the image encoder of a pretrained CLIP model; for text, the hidden states of a small pretrained language model. The choice matters and we will return to it, but the shape of the method does not depend on it.

Normalise every embedding to unit length:

Why normalise? Because otherwise vector length becomes a confound. The raw dot product e·e′ = ‖e‖‖e′‖cosθ rewards long vectors for being long. Encoders systematically give longer vectors to some kinds of input — high-contrast images, long captions — and you do not want "this photo is vivid" to read as "this photo is a duplicate of everything." Normalising strips length and keeps only direction, so the only surviving quantity is the angle:

Two examples are semantic duplicates if that cosine exceeds a threshold ε. The whole method is now specified: compute all pairwise cosines, group anything above ε, keep one per group. And it is completely impossible to run.

Step 2: the quadratic wall, with real numbers

SemDeDup's LAION experiments used a subset of about 440 million pairs. The number of distinct pairs is:

Ninety-seven quadrillion pairs. Two ways to feel how bad that is.

Memory. Storing that similarity matrix at 2 bytes per entry (half precision) needs 9.68 × 10¹⁶ × 2 = 1.94 × 10¹⁷ bytes, which is 193 petabytes. There is no machine.

Compute. Each cosine of two 512-dimensional unit vectors is 512 multiplies and 511 adds, call it 1,024 floating-point operations. So:

A GPU sustaining 10¹⁴ FLOP/s on this kind of dense arithmetic would need 9.9 × 10⁵ seconds — about 11.5 GPU-days of nothing but dot products, before you have looked at a single duplicate. And you would still have nowhere to put the answers.

Step 3: cluster first, and count what that saves

The insight is almost embarrassingly simple, and it is the same insight behind every approximate nearest-neighbour index: duplicates are, by definition, close together, so you never need to compare things that are far apart.

Run k-means on the embeddings with k clusters. Then only compare items that landed in the same cluster. If the clusters are roughly balanced, each holds N/k items, and the pair count becomes:

Compare to N²/2 for the full matrix: the saving is exactly a factor of k. Not approximately — exactly, under the balanced-cluster assumption. That is a clean result worth holding on to: the number of clusters is the speedup.

SemDeDup used k = 50,000 on the LAION subset. Run the arithmetic:

From 193 petabytes to 155 megabytes, and from a week and a half of GPU time to twenty seconds. And the 155 MB figure is per cluster, processed one at a time, so the job is trivially parallel and embarrassingly memory-friendly. That table is the whole reason SemDeDup can be run by a normal team on a normal cluster.

Step 4: what the approximation costs, honestly

Restricting comparisons to within-cluster means SemDeDup will miss any duplicate pair that k-means split across two clusters. Is that a lot?

Think about the geometry. Two semantic duplicates have cosine near 1, which means they are nearly the same point on the unit sphere. For k-means to separate them, the Voronoi boundary between two centroids would have to pass almost exactly between them. The set of points within cosine 0.99 of each other that straddle a boundary is small — it is essentially the surface area of the boundaries times the duplicate radius.

The simulation later in this chapter lets you toggle between cluster-scoped and global comparison on a small set so you can see exactly which pairs the clustering misses, and how few they are.

Step 5: which duplicate survives?

Now the decision that people skip and that turns out to matter. You have found a group of, say, three items that are all above threshold with each other. You keep one. Which?

Three defensible rules:

The counterintuitive rule wins, and the reason is the same distribution argument from Chapter 1. If you always keep the prototype, then after aggressive dedup your dataset is a list of cluster centres: exactly the head of the distribution, exactly the part you already had too much of. Keeping the outlier of each duplicate group preserves the corners of the space.

The three-vector worked example

Do it by hand. A cluster contains three near-identical product photographs of the same white sneaker. Their embeddings, in a toy three-dimensional space whose axes you can read as "sneaker-ness", "white-background-ness", and "studio-lighting-ness":

First check they are unit length, because every step below assumes it. For A: 0.60² + 0.48² + 0.64² = 0.3600 + 0.2304 + 0.4096 = 1.0000. Square root is 1. B and C are permutations of the same three components, so they are unit too. Good — cosine is now just the dot product.

Pairwise cosines. Multiply component by component and add:

With ε = 0.95, all three pairs are above threshold. They form one connected group: A–B, A–C, and B–C. Two survive-or-die decisions to make, and we keep exactly one of the three.

The cluster centroid. SemDeDup uses the k-means centroid of the whole cluster, but for a three-member cluster that is just their mean. Average each component:

The centroid of unit vectors is not unit, so we need its length before taking cosines:

Cosine to the centroid, for each candidate. Dot with c, then divide by ‖c‖:

The decision. Rank by cosine to the centroid: A (0.998759) is the most central, then B (0.996617), then C (0.992333). So:

Notice how narrow the differences are: 0.9988 versus 0.9923, a gap of 0.0065. On any single group the choice is nearly arbitrary. Across a hundred million groups it is the difference between a dataset that keeps its shape and one that shrinks toward its own average.

The algorithm, complete

python — SemDeDup, end to endimport numpy as np

def semdedup(embeddings, k, eps, keep="farthest"):
    """embeddings: (N, d) float32.  Returns a boolean keep-mask of length N."""
    E = embeddings / np.linalg.norm(embeddings, axis=1, keepdims=True)

    # --- one linear pass: assign every point to a centroid ---
    km       = kmeans(E, k)                 # faiss.Kmeans in practice
    labels   = km.labels_                    # (N,)
    centroid = km.cluster_centers_           # (k, d), NOT unit length

    keep_mask = np.ones(len(E), dtype=bool)

    for c in range(k):                      # each cluster is independent -> parallel
        idx = np.where(labels == c)[0]
        if len(idx) < 2: continue
        Ec  = E[idx]                         # (m, d), m ~ N/k

        # rank members by how central they are in THIS cluster
        cen  = centroid[c] / np.linalg.norm(centroid[c])
        csim = Ec @ cen                      # (m,) cosine to centroid
        # the FIRST item in the order is the one that survives its group,
        # so "keep farthest" means process LEAST-central first. Easy to write backwards.
        order = np.argsort(csim) if keep == "farthest" else np.argsort(-csim)
        Ec, idx = Ec[order], idx[order]

        S = Ec @ Ec.T                        # (m, m) -- 155 MB at m = 8800, fp16
        # upper triangle: "does an EARLIER (more central) survivor already cover me?"
        S = np.triu(S, k=1)
        covered = (S > eps).any(axis=0)     # (m,) True -> drop
        keep_mask[idx[covered]] = False

    return keep_mask

Read the triu line carefully, because it is where the survivor rule is actually implemented and it is easy to get backwards. Items are sorted most-central first. S > eps in the strict upper triangle asks, for each item, "is there an item earlier in the order that I duplicate?" If yes, this item is dropped. So the item that survives a group is the one with no earlier duplicate — the first in the ordering. Sorting most-central first therefore keeps the most central. To keep the farthest, you sort least-central first, which is what the argsort(-csim) versus argsort(csim) switch does.

The simulation: watch the survivors

Now play with it. Below is a two-dimensional stand-in for embedding space. The points are unit vectors on a high-dimensional sphere, drawn here by a projection with one honest property: for L2-normalised vectors, squared Euclidean distance and cosine are exactly related by

The plot is scaled so that a distance D across the box corresponds to cosθ = 1 − D². So the threshold is literally a radius you can see — which is why the slider below is calibrated in radius, with the corresponding ε displayed beside it. A radius of 0.03 is ε = 0.9991; a radius of 0.15 is ε = 0.9775.

Three things worth doing before moving on. Pull the radius down to 0.01 and notice that almost nothing is removed: at that threshold only the tight planted knots are caught. Push it to 0.12 and watch entire clusters collapse to two or three survivors — that is over-deduplication, and the removed points are no longer duplicates in any meaningful sense. And flip the survivor rule at a moderate threshold and watch the survivors move: with "closest," they migrate inward toward the cluster cores; with "farthest," they stay on the rim.

What the paper actually measured

Now the results, and they deserve to be stated with their caveats attached rather than as a slogan.

On the LAION image-text subset, SemDeDup with an appropriate threshold removed roughly half the pairs. A CLIP model trained on the deduplicated half, with the same number of training iterations, matched the model trained on the full set on in-distribution evaluation — and did slightly better on out-of-distribution evaluations. Reaching a target accuracy took roughly half the training iterations, because half the iterations were no longer being spent on repeats.

On text, the same procedure applied to a large web corpus with embeddings from a small pretrained language model let them remove a much smaller fraction — on the order of 15% — with no loss in perplexity and a proportional reduction in training time, plus improved perplexity on out-of-distribution text.

Why out-of-distribution performance goes up

The most surprising result deserves its own explanation, because "remove data, get better" trips people's alarms and it should.

Return to Chapter 1's arithmetic. A duplicate is an unrequested importance weight. Which examples get duplicated most on the web? The popular ones — the head of the distribution. So the duplication pattern is not uniform noise; it is a systematic upweighting of exactly the region your model already handles well.

Stated as one sentence: deduplication is distribution flattening disguised as compression. The compute saving is the advertised benefit. The distributional correction is the real one.

What SemDeDup is not

Two boundaries, because both get crossed constantly in practice.

It is not a quality filter. SemDeDup removes redundancy, not junk. A corpus of ten million distinct pieces of keyword-soup alt-text will pass through SemDeDup almost untouched, because junk is often highly diverse — that is what makes it junk. Conversely a corpus of pristine, expert-written captions about one narrow topic will be heavily deduplicated. The two axes are close to orthogonal.

It is not a difficulty or influence metric. There is a separate literature on pruning by how much an example contributes to learning — forgetting scores, gradient-norm ranking, influence functions, and the memorisation-based metrics. Those ask "does the model need this?" SemDeDup asks the cheaper, more robust question: "do I already have this?" The second question requires no training run to answer, which is precisely why it scales.

The third column explains the adoption pattern. Influence-based pruning is more principled and it is barely used at web scale, because it requires the training run you were trying to make cheaper. Embedding-based curation is a strictly worse approximation that costs a single forward pass, and that ratio is why it won.

Choosing k, and what going wrong looks like

The number of clusters is the only free parameter besides the threshold, and both failure directions are instructive.

The rule of thumb that balances these is k ≈ √N, which makes cluster size also ≈ √N and equalises the two costs (the linear assignment pass and the quadratic within-cluster pass). For 440M that would suggest about 21,000 clusters; the paper used 50,000, which pushes cluster size down to 8,800 and buys a smaller memory peak at slightly higher clustering cost. Both are in the sensible band, and the result is not sensitive within it.

Thresholds do not transfer, and here is the proof

One last warning before the results. The value of ε that removes 50% of a LAION subset with a particular CLIP image tower will remove some entirely different fraction of your corpus with a different encoder, because the two models produce different similarity distributions.

Chapter 3's random-cosine argument gives the intuition: the spread of similarities depends on the effective dimensionality of the representation, and the location of the "duplicate" mode depends on what the encoder was trained to consider identical. Two encoders can agree perfectly on the ordering of pairs and disagree completely on the numerical threshold that separates duplicates from neighbours.

Third worked example: a six-item cluster, step by step

The three-vector example showed the survivor rule. This one shows the greedy pass that implements it, because the mechanics have a subtlety worth catching.

A cluster of six items, with these cosines to the cluster centroid:

Suppose the pairwise cosines put i₁, i₂, i₃ all above ε = 0.95 with one another, i₄ and i₅ above threshold with each other, and i₆ above threshold with nothing.

Run it with the SemDeDup rule — farthest survives, so process least-central first: i₆, i₅, i₄, i₃, i₂, i₁.

Now run it with the opposite rule — closest survives, so process most-central first: i₁ keeps and absorbs i₂ and i₃; i₄ keeps and absorbs i₅; i₆ keeps. Survivors: i₁, i₄, i₆.

Same count — three — different members. The removal rate is identical, so a dashboard reporting only "50% removed" cannot distinguish the two runs. The difference is entirely in which three survive, and that is what shows up months later as a model that is good at prototypes and bad at edges.

The subtlety: duplicate groups are not always cliques

Everything above assumed the duplicate relation is transitive within a group. It is not, and the failure is instructive.

Consider three items with:

At ε = 0.95, a and b are duplicates, b and c are duplicates, but a and c are not. This is a chain, not a clique, and it is extremely common: near-duplication is a similarity relation, and similarity relations are not transitive.

Now watch the greedy pass produce two different answers depending on order:

One order keeps two items, the other keeps one. Neither is wrong — there is no ground truth about how many distinct things a chain contains — but it means the ordering rule is doing two jobs, not one: it decides which member of a group survives, and it partly decides how the groups are drawn in the first place.

Chapter 3: What a Hash Cannot See

Every data pipeline already deduplicates. Ask an engineer and they will tell you they hash the files and drop collisions. So a fair challenge to Chapter 2 is: what exactly does an embedding buy that a hash does not, and is it worth a GPU pass over the corpus?

To answer that properly we have to build a perceptual hash from zero, by hand, and find its edges.

The duplicate ladder

Rungs 1 to 3 are the pixel-level regime. Rungs 4 to 6 are the semantic regime, and the jump between them is the jump this chapter is about.

Building a perceptual hash by hand

The simplest useful perceptual hash is the difference hash, dHash. The recipe is four lines of prose:

Step 2 is the clever one. By storing only relative brightness, the hash becomes invariant to anything that shifts or scales brightness uniformly. Let us prove that on real numbers.

Worked example: 16 bits by hand

Take a 5-wide, 4-tall greyscale patch (which gives 4 comparisons per row, 16 bits total — a miniature dHash you can compute in your head):

Test 1: brighten the whole image by 20. Row 1 becomes 30, 60, 50, 110, 100. Comparisons: 30>60 no, 60>50 yes, 50>110 no, 110>100 yes → 0101. Identical. And it will be identical for every row, because adding a constant to both sides of an inequality never changes the inequality. Hamming distance 0.

That is a genuine, provable invariance, and it is why perceptual hashing works at all. Exposure changes, gamma shifts, and mild recompression all move pixel values without reordering neighbouring pixels.

Test 2: crop one column from the left. Now each row is shifted, with a new column arriving on the right. Row 1 becomes 40, 30, 90, 80, 75:

Seven bits out of sixteen. Scaled to the standard 64-bit hash that is about 28 differing bits, against a duplicate threshold of 5. The crop is rejected. Same photograph, ten pixels narrower, and the hash calls it a different image.

Test 3: a second photograph of the same sneaker, taken from a slightly different angle. Every pixel is different. The dHash bits are effectively a fresh random 64-bit string. Expected Hamming distance to the original: 32 of 64. This is not a near-miss — it is indistinguishable from comparing two unrelated images. A perceptual hash is not merely bad at rung 4 of the ladder; it has no signal there at all.

The text side: MinHash, and the S-curve

Text pipelines use a different tool with the same shape, and it is worth deriving because the tradeoff it exposes is exactly the one an embedding threshold faces.

Represent a document as the set of its overlapping word n-grams (shingles). Similarity between documents is the Jaccard index:

The MinHash trick: apply a random permutation to the universe of shingles and record the minimum element of each set. The probability that two sets have the same minimum is exactly J. That single fact turns a set-overlap computation into a coin flip you can repeat.

Now the indexing scheme. Take r · b hashes, arrange them in b bands of r rows, and declare two documents candidates if they match on all r hashes of any band. The probability of being flagged is:

Work it with r = 5 and b = 20. For a truly near-duplicate pair with J = 0.8:

And for a merely related pair with J = 0.3:

Eighty-percent-similar documents are caught 99.96% of the time; thirty-percent-similar documents are caught 4.75% of the time. That is a beautifully sharp threshold, produced entirely by choosing r and b, and it is the reason MinHash-LSH is the backbone of every large text-dedup pipeline.

The simulation: three verdicts, one image

Walk the buttons in order and watch the two verdicts diverge. "Brighter" is the case the hash was designed for: zero differing bits. "Cropped" is the case that breaks it: the picture is obviously the same and the hash says no. "Re-shot" is the case the hash cannot even attempt. And the last two are the interesting ones, because they are where the embedding starts making decisions you have to think about.

The threshold is a policy, not a fact

Two photographs of the Eiffel Tower taken by different tourists on different days will sit at a high cosine — plausibly above 0.95 in a CLIP-style space. Are they duplicates?

It depends entirely on what you are training. For a general vision-language model, ten thousand near-identical Eiffel Tower shots are redundancy and you should keep a handful. For a landmark-recognition system, that variation across lighting, season, and viewpoint is the training signal and deleting it is self-harm. The cosine is the same number in both cases. Only the decision differs.

So calibrate, and calibrate against something real. Here is what that looks like on a small labelled sample: take 200 random pairs from your corpus that are above cosine 0.80, label 20 of them by hand as duplicate or not, and count.

Check the F1 at ε = 0.85 by hand: 2 · (0.643 · 0.900) / (0.643 + 0.900) = 2 · 0.5787 / 1.543 = 1.1574 / 1.543 = 0.750. At ε = 0.95: 2 · (1.000 · 0.600) / 1.600 = 1.200 / 1.600 = 0.750. The middle threshold wins on F1.

What the embedding actually encodes, and where it came from

One more thing to be precise about, because it determines which duplicates your pipeline can see. A CLIP-style image embedding is invariant to roughly what its training augmentations and its training data made it invariant to. Random-resized-crop during training teaches it to ignore framing. Colour jitter teaches it to ignore white balance. Contrastive training against captions teaches it to ignore anything a caption would not mention.

That last clause is the load-bearing one. The embedding's notion of "same" is inherited from the supervision that built it. If captions never distinguish two things, the embedder will not either, and your deduplicator will treat them as duplicates.

That last row became a small research finding of its own: a substantial share of high-scoring web image-text pairs score highly for a boring reason — the caption is a transcription of text visible in the image. Masking the rendered text before scoring changes which pairs a filter keeps, and improves the resulting model. Hold that thought; it returns in Chapter 5 as one of the ways a leaderboard teaches you something the metric alone would not.

Choosing an embedder by its invariances

"Use an embedding" is not advice. Which embedding decides which duplicates you find and which distinctions you destroy. Here is the family tree, ordered from most semantic to most literal.

Read the invariance column top to bottom and you are reading a dial from "same meaning" to "same origin." Those are different questions, and a pipeline usually needs answers to both, from different models.

What each stage costs, and why the pipeline has two

Almost every production pipeline runs a cheap syntactic pass followed by an expensive semantic one. The arithmetic explains why.

Exact hashing over a billion files is a checksum per file. It is disk-bound, not compute-bound, and it costs essentially nothing beyond the read you were doing anyway.

Perceptual hashing is a decode, a resize to 72 pixels, and 64 comparisons. Thousands of images per second per CPU core. A billion images is a few hundred core-hours — a rounding error.

Embedding is a neural forward pass. At a few thousand images per second on a single accelerator, a billion images is on the order of a hundred GPU-hours — a day on a small node, and real money at billion scale.

So you run the cheap ones first, and not only to save money. The cheap pass also shrinks the expensive one. If perceptual hashing removes 12% of the corpus, the semantic stage's within-cluster comparison count — which is quadratic in cluster size — falls by:

Twelve percent off the item count buys twenty-three percent off the quadratic. That leverage is the general reason cheap filters go first, and it applies to every stage ordering decision in Chapter 8.

Reading a cosine histogram

Before you pick any threshold, plot the distribution of nearest-neighbour cosines across a random sample. It is the single most informative picture in this whole lesson, and it usually has three distinguishable regions.

If your histogram has a clean spike separated from the shoulder by a valley, you have been handed your threshold: put it in the valley. If it does not — if the density decreases smoothly all the way to 1.0 — then there is no natural notion of "duplicate" in your corpus under that embedder, and every threshold is a policy decision you are making rather than discovering. Both outcomes are useful to know, and you cannot know either without the plot.

Worked example: one word, and Jaccard falls off a cliff

The MinHash section gave the indexing mathematics. Here is the underlying similarity computed by hand, because the number is more shocking than the formula suggests.

Two sentences:

Shingle each into overlapping word triples. S₁ has six words, so four triples:

One word changed out of six, and the Jaccard index reads 0.33 — the same value you would get from two documents sharing a third of their content. Now push that through the banding formula with r = 5 and b = 20:

Under 8%. The index will miss this pair more than nine times out of ten. Meanwhile a sentence embedder would place these two at a cosine somewhere around 0.97, and any semantic deduplicator would catch them immediately.

Reading the three thresholds side by side

One table to hold the whole chapter. Same pair of items, three instruments, three answers — and each answer is correct for a different question.

Rows four and five are the ones to sit with. In both cases the embedding says "duplicate" and the correct action differs, because the question being asked differs. No threshold can distinguish them, because the distinction is not in the data — it is in what you intend to build.

Two more instruments, and where they sit

The ladder in this chapter had two rungs of tooling — hashes and embeddings. Two more show up in real pipelines and it is worth knowing where they fall.

The second row is a common source of confusion. Compressing an embedding to 64 bits does not turn a semantic comparison into a perceptual one; it is the same similarity, approximated. Whereas a perceptual hash computes a genuinely different quantity. Same output type — a short bit string with a Hamming distance — and completely different meaning.

The same ladder in other modalities

Read the last column across all four. Every modality has a case where "these two items are similar" and "these two items are redundant" come apart for a reason specific to that medium. There is no general answer, and the video row — where near-duplication is the natural state of the data — is the clearest warning that the method in Chapter 2 assumes a corpus made of knots, not of continua.

Chapter 4: Benchmark the Filter

Chapter 2 gave us a curation method. It did not give us a way to know whether it was any good, and that gap is larger than it sounds.

Suppose you invent a filter. You apply it, train a model, and get 61% on your evaluation. Is that good? Compared to what? You could train an unfiltered baseline — but then you have used more data, so the comparison is confounded by dataset size. You could match the dataset size by randomly subsampling — better, but now your compute differs from every published number, so you cannot compare to anyone else's filter. And if a competing filter is evaluated by a different lab with a different learning-rate schedule, the comparison is meaningless.

The inversion

Standard machine-learning benchmarks fix the data and let you vary the model. ImageNet hands you 1.28 million labelled images and asks for the best classifier. GLUE hands you nine tasks and asks for the best language model.

DataComp does the exact opposite. It fixes the model, the training code, the hyperparameters, the compute budget, and the evaluation suite. The only thing you are permitted to change is which subset of a fixed pool you train on.

Sit with how strange that is. A DataComp submission is not code that runs at inference. It is not weights. It is a set of universally unique identifiers — a list saying "train on these." Everything else in the pipeline is somebody else's frozen decision.

The pieces, and why each one is shaped the way it is

CommonPool. 12.8 billion image-text pairs extracted from Common Crawl, released as the shared substrate. Note the word pool, not dataset: it is deliberately unfiltered beyond safety processing, because a pre-cleaned pool would bake the organisers' filtering opinions into everyone's starting point.

Four scales. Filtering behaviour is not scale-invariant, so the benchmark runs at four sizes, each a factor of ten apart:

The small scale is deliberately runnable by a research group with a handful of GPUs, which is the design decision that made the benchmark actually get used. A benchmark only the largest labs can enter does not produce a field.

Samples seen equals pool size. This is the single most important design choice in the whole benchmark, and it is easy to skim past. The compute budget is measured in total samples processed, and it is set equal to the size of the unfiltered pool, regardless of how much you filter away.

Why "samples seen" and not "epochs"

Imagine the alternative. Suppose the budget were "one epoch over whatever you kept." Then a filter that removes 90% of the pool trains for 90% less compute. Its resulting model would probably be worse, and you would have learned nothing about the filter, because the comparison confounded data quality with training length.

Worse, the incentive would be inverted: the winning "filter" would be the identity function, since keeping everything buys the most compute.

Fixing samples-seen removes the confound and creates the real tradeoff. Work the arithmetic at the medium scale, where B = 128 million:

Now the curve has to be unimodal, and it is: quality rises monotonically as you filter harder, diversity falls monotonically, and the product has an interior maximum. The benchmark's job is to let people find where it is.

The evaluation suite, and why it is 38 tasks

The benchmark evaluates zero-shot on 38 downstream datasets: ImageNet and several of its distribution-shifted variants, a broad transfer collection covering natural, specialised, and structured domains, plus image-text retrieval benchmarks.

Why so many? Because a single evaluation number invites the field to overfit to it, and in a data benchmark that failure mode is unusually easy to hit. If ImageNet accuracy were the only score, the optimal strategy would be "retrieve the nearest neighbours of ImageNet training images from the pool and train on those" — which is a real and very effective strategy, and also not a general-purpose curation method. Reporting a 38-task average makes narrow benchmark-shaped filters visible as a gap between two columns.

Two tracks

The two tracks answer different questions and both matter. The filtering track isolates the algorithm. The bring-your-own-data track admits that in real life you also get to choose where the data comes from, and lets external corpora compete.

What the benchmark cannot tell you

A benchmark's limitations are part of its specification, and DataComp is unusually clear about its own. Four worth internalising before you cite any number from it.

Recipe dependence. Every result is "best filter for this training recipe at this compute budget." A filter that wins at 128 million samples seen with a ViT-B/32 need not win for a different architecture, a different loss, or a ten-times-longer run. The benchmark controls the confound by freezing it, which means the conclusion is conditional on the frozen value.

Filtering is only one intervention. The track's rules permit choosing rows and forbid changing them. But some of the largest gains in the following two years came from rewriting data — generating synthetic captions for images whose alt-text was useless, for instance. That whole family of methods is outside the filtering track by construction.

Anchor contamination. Several strong baselines use ImageNet as a reference set to decide what to keep. When you then evaluate on ImageNet, part of the measurement is circular. The paper is upfront about this; readers frequently are not. Chapter 7 makes this into a general principle about feedback loops.

A web pool is a web pool. CommonPool inherits everything that is in Common Crawl, including material that people did not consent to have used and content that safety processing catches imperfectly. Face blurring and NSFW filtering were applied; they are mitigations, not solutions. Any serious engagement with web-scale curation has to hold this alongside the technical results rather than after them.

A note on what "same" means here

One phrase does a lot of work in this chapter and deserves pinning: "identical training." It means the same source revision of the training script, the same hyperparameters, the same random seed, the same hardware precision, and the same evaluation code — not merely the same architecture name and learning rate.

Those extra clauses are not pedantry. Precision changes, dataloader shuffling, and evaluation-script versions each move results by amounts comparable to the effects being measured. A benchmark that specifies the model but not the script has not frozen anything; it has just moved the unmeasured variance somewhere less visible.

What a submission looks like on disk

Abstractions become concrete when you see the file. A filtering-track entry is, literally, this:

python — the whole of a DataComp filtering submissionimport numpy as np, pandas as pd

# The organisers ship, per shard: uid, plus PRECOMPUTED CLIP embeddings.
# You never touch a pixel unless you want to.
meta = pd.read_parquet("shard_00042.parquet")      # uid, url, text, ...
img  = np.load("shard_00042_img_emb.npy")          # (n, 768) float16
txt  = np.load("shard_00042_txt_emb.npy")          # (n, 768) float16

# --- your entire contribution starts here ---
img = img / np.linalg.norm(img, axis=1, keepdims=True)
txt = txt / np.linalg.norm(txt, axis=1, keepdims=True)
score = (img * txt).sum(axis=1)                  # (n,) the CLIP score
mask  = score > THRESHOLD                            # global 70th percentile
# --- and ends here ---

np.save("my_subset_uids.npy", meta.uid.values[mask])

# Then: `python train.py --scale medium --data_dir ... --uids my_subset_uids.npy`
# and `python evaluate.py --track filtering`. You changed nothing else.

Six lines of numpy between two comment markers. That is the artifact. Everything else — the model, the optimiser, the schedule, the resampling, the 38 evaluations — is somebody else's frozen decision, identical for every entrant.

How training reaches the budget from a smaller set

One mechanical detail that trips people up. If you keep 38.4 million pairs and the budget is 128 million samples seen, the training loop does not run 3.33 "epochs" in the tidy sense. It resamples with replacement from your shards until the counter hits 128 million.

Two consequences worth knowing:

• The number of times a given example is seen is not exactly 3.33; it is a Poisson-ish spread around it. Some examples are seen twice, some six times. At a mean of 3.33 the variance barely matters, but at a mean of 33 it starts to.
• Batch composition still comes from your filtered set, so the contrastive negatives are drawn from your distribution. A filter that concentrates the data also makes the negatives harder — a second-order effect nobody controls for, and a plausible partial explanation for why aggressive filtering degrades faster than the pure diversity argument predicts.

What the 38 evaluations cover

Notice that the suite is deliberately unbalanced toward things a naive filter will not improve. That is a feature. A benchmark whose tasks all reward the same intervention cannot distinguish between two interventions.

Three ways benchmarks fail, and how this one avoided them

The cost of entry, and why that number matters

The four-scale structure encodes a research methodology: search cheaply at small, confirm at medium, verify the trend at large, and only then spend real money. And it exposes something the field needed to know — that filter rankings are not stable across scales, so a method validated only at small scale is a hypothesis, not a result.

The counterfactual: what a field looks like without the instrument

It is worth spelling out what was actually broken, because the same breakage exists in most engineering organisations right now and is rarely named.

Read the left column and ask which of those sentences describes a decision your own team makes. Sampling policies, retrieval corpora, label pipelines, evaluation sets — all of them are usually changed without a controlled comparison, for exactly the reason data was: building the control is boring and nobody gets credit for it.

What to demand of any data claim, including your own

Where this benchmark sits among data-centric efforts

Each row trades attribution against realism. The filtering benchmark sits at the attribution end deliberately, and its limitations are the price of that position rather than oversights.

The five design choices, ranked by how much they mattered

Benchmarks are mostly design decisions. Here they are in order of consequence, which is a useful exercise to run on any benchmark you are asked to trust.

Notice that four of the five are about removing degrees of freedom rather than adding capability. That is what benchmark design is: deciding what participants are not allowed to change, so that what they do change becomes measurable.

What a bad version of this benchmark would look like

Concretising the failure modes makes the design choices feel earned.

The evaluation harness, in shape

Concreteness again: the other end of the pipeline is as fixed as the training end, and seeing it makes the "38 numbers" concrete.

python — the frozen evaluation side, in outlinedef evaluate_all(model, tasks):
    results = {}
    for t in tasks:                              # 38 of them
        # Class names -> prompts -> text embeddings. The classifier is BUILT
        # from language at evaluation time; no head is ever trained.
        W = build_zeroshot_classifier(model, t.classnames, t.templates)
        correct = total = 0
        for images, labels in t.loader:
            feats = model.encode_image(images)
            feats = feats / feats.norm(dim=-1, keepdim=True)
            preds = (feats @ W).argmax(dim=-1)
            correct += (preds == labels).sum().item(); total += len(labels)
        results[t.name] = correct / total
    results["average"] = sum(results.values()) / len(results)
    return results                              # 38 numbers + the mean

Two details worth catching. First, the classifier is constructed from class names at evaluation time — no head is trained on any downstream task, so nothing about the evaluation can leak into the model. Second, the average is an unweighted mean over tasks with different metrics and different chance levels, which makes its absolute value close to meaningless while leaving comparisons between entries perfectly valid, since every entry is aggregated identically.

One more thing the freeze bought

A subtle benefit of freezing the recipe: it made negative results publishable. If your clever filter loses to a one-line cosine threshold, that is now a citable fact rather than a private disappointment, because the comparison is unambiguous and anyone can reproduce it.

Fields without a shared instrument accumulate positive results only, which means every practitioner independently rediscovers the same dead ends. A leaderboard where obvious-but-worse ideas are visibly obvious-but-worse is doing as much work as the winning entry, and it is the part that never gets credited.

The BYOD track, and why it exists

A benchmark that only permits filtering a fixed pool answers one question well and quietly forbids another: is filtering a web pool even the right strategy? The bring-your-own-data track exists to keep that question open.

The design tension is honest and unresolved: the filtering track is rigorous and narrow, the open track is realistic and confounded, and a field needs both. Running them side by side under one compute budget is the compromise, and it lets a reader see how far pure selection gets before external data has to enter.

Chapter 5: What Actually Wins

With the instrument built, the question becomes empirical. Somebody ran the baselines. Here is what they found, in the order that a person discovering it would find it, including the parts that contradict the tidy story.

The score everything is built on

The workhorse filter is CLIP score, and it is one line. Take a pretrained CLIP model. Embed the image with its image tower, embed the caption with its text tower, normalise both, take the dot product:

That number is a direct measurement of the quantity that matters: does this caption describe this image? Chapter 1 argued that mismatched pairs are the expensive failure. CLIP score is a cheap estimator of mismatch, and it needs no training, no labels, and one forward pass per modality.

Rank the pool by s and keep the top fraction. Worked example on a pool of ten pairs, sorted:

Look at rank 4. "Nike Air Max 90 white size 10 mens" is true, and useful, and it gets dropped by a top-30% cut. That is the cost of the filter made visible: it is not separating true from false, it is ranking by descriptiveness in CLIP's own idiom, and terse product metadata reads as low-descriptiveness even when accurate.

The baseline table

These are the reported filtering-track baselines at the medium scale — a 128-million pool with a 128-million-sample budget. Two columns, because they disagree.

Treat the ordering and the size of the gaps as the lesson, not the third decimal place; exact values shift with scale and with which evaluation subset is averaged.

Finding 1: a one-line filter nearly doubles accuracy

No filtering gives 0.176 on ImageNet. CLIP score with an L/14 scorer at top 30% gives 0.273. That is a relative improvement of 55% from a rule you can write in an afternoon, with no training.

It is worth pausing on how much of the field's effort would be needed to find 10 accuracy points from architecture at this scale. Years. The data axis had been sitting there untouched because nobody could measure it.

Finding 2: aggressiveness has an optimum, and it is not at the extremes

Sweep the keep fraction with the CLIP-score filter and the curve rises, peaks, and falls. At the medium scale, top-30% is around the peak. Both directions from there are worse, for the two reasons Chapter 4 set up: keep more and the noise comes back; keep less and the epoch count climbs into memorisation.

And the optimum moves with scale. At larger pools you can afford to be more aggressive, because 10% of 1.28 billion is still 128 million unique pairs — plenty of diversity — whereas 10% of 12.8 million is 1.28 million, which is not. The keep fraction is not a constant of nature; it is a function of how many unique examples you end up with relative to your budget.

Finding 3: combining an alignment filter with a distribution filter beats either

The two strongest individual baselines measure different things. CLIP score asks is this pair aligned? The image-based filter asks a completely different question: does this image look like the kind of image we care about? It works by clustering image embeddings and keeping clusters whose centres are near ImageNet training images.

Their intersection — keep pairs that are both well-aligned and in a desirable cluster — gives the best ImageNet result, 0.297. Two filters attacking orthogonal failure modes compose, exactly as Chapter 1's taxonomy predicts: one attacks misalignment, the other attacks distribution skew.

Finding 4: the winner depends on the yardstick, and this is the important one

Read the last two rows of the table again, carefully.

The intersection filter is better on ImageNet by 2.4 points and worse on the 38-task average by 1.0 point. There is no single winner, and the reason is not subtle: the image-based half of the intersection is anchored on ImageNet. It keeps images that look like ImageNet images. So it optimises ImageNet accuracy, and it does so partly by narrowing the dataset in a direction that costs you on the other 37 tasks.

Finding 5: a better scorer makes a better filter — but "better" is its own axis

Compare the two CLIP-score rows: swapping a B/32 scorer for an L/14 scorer moves ImageNet from 0.256 to 0.273 with the identical filtering rule. The filter is only as good as the embedding underneath it, which is the opening of Chapter 7's feedback loop.

But then a follow-up result complicated this in a genuinely surprising way. If you stop treating the filter model as a hand-me-down and instead train a network whose only job is to filter, you get much better datasets — and the crucial finding is that filtering ability and classification ability come apart. A network that is a worse zero-shot classifier can be a better data filter. The two capabilities are correlated but distinct, and optimising the wrong one wastes effort.

The bias CLIP score carries, and what to do about it

CLIP score is not a neutral measurement of "is this a good pair." It measures agreement in a particular learned space, and that space has a shape.

The rendered-text bias got measured directly. If you mask the text regions of an image before computing the score, a substantial share of high-scoring pairs collapse — their alignment was carried entirely by the words in the picture. Filtering on the masked score keeps a different, better set. That result is a good template for interrogating any scorer: ablate the shortcut and see how much of the score was the shortcut.

The simulation: compare the filters yourself

Drag the keep-fraction slider to 100% and to 3% and read the annotation: at one extreme you are seeing 128 million noisy pairs once each, at the other you are seeing 3.8 million clean pairs thirty-three times each. Neither is where the peak is. The peak is where the marginal removed example stops being noise and starts being coverage.

The whole chapter as a decision procedure

Why CLIP scores are small numbers, and what a threshold of 0.28 means

The first time you compute CLIP scores you notice something odd: matched pairs score around 0.30, mismatched pairs around 0.10, and almost nothing reaches 0.5. If cosine runs to 1.0, why is everything crowded into the bottom third?

Two reasons, and both are worth understanding because they determine how to set thresholds.

Reason 1: random vectors in high dimensions are nearly orthogonal. Take two independent random unit vectors in d dimensions. Their cosine has mean zero and standard deviation:

For d = 512 that is 1/22.63 = 0.0442. So a cosine of 0.30 sits about 6.8 standard deviations above chance. It is an enormous signal that merely looks like a small number, because our intuition for cosine comes from two and three dimensions where random vectors are often quite aligned.

Reason 2: the modality gap. In a contrastively trained two-tower model, image embeddings and text embeddings do not interleave. They occupy two separate cones on the sphere, with a gap between them that training never closes — because the loss only cares about relative ranking within a batch, and a uniform offset between the two clouds ranks identically. So the achievable cross-modal cosine has a ceiling well below 1.0 that has nothing to do with how well the pair matches.

Setting a percentile threshold from a sample

You cannot sort a 12.8-billion-item pool to find its 70th percentile without a very annoying amount of I/O. You do not have to. Sample.

Draw a uniform random sample of n items, sort those, and take the sample's 70th percentile as your threshold. The standard error of a sample quantile at probability p is approximately:

where f(q) is the density of scores at the quantile. The first factor is the one you control. With p = 0.7:

Check the first row: √(0.7 × 0.3 / 1000) = √(0.21/1000) = √0.00021 = 0.0145. A sample of a hundred thousand is already overkill for a decision whose optimum is a broad plateau. Do not build infrastructure to compute an exact quantile over a pool; sample a hundred thousand rows and move on.

Which filters compose, and which just fight

Finding 3 said that intersecting an alignment filter with a distribution filter beats either. That is not a general law about intersections, and knowing when composition helps is worth more than the specific result.

The rule underneath: filters compose when they are attacking different failure modes and fight when they are correlated estimators of the same one. Before adding a filter, ask which of Chapter 1's three failures it addresses. If the answer is one you already cover, you are buying correlation, not coverage.

The other track, and the finding it produced

The bring-your-own-data track let entrants assemble a training set from anywhere, subject to the same compute budget. Two results are worth carrying.

First, well-curated external corpora are strong — unsurprising, and it establishes a useful upper reference for how far pool filtering can go.

Second, and more interesting, the thing that eventually beat careful filtering was not better selection at all. It was rewriting: generating synthetic captions for images whose alt-text was useless, so that a pair which any alignment filter would have discarded becomes a good pair instead. A discarded image is a wasted image; a recaptioned one is recovered inventory.

Running this leaderboard on your own problem

The benchmark's structure ports to any system where you control a dataset. Five steps:

Worked example: the cut, and what it bought

Take the ten-pair table above as a miniature pool and do the whole calculation, so the abstract "top 30%" becomes arithmetic.

Finding the cut. Ten items sorted descending. A top-30% cut keeps the highest 3, so the threshold sits between the 3rd and 4th scores:

What it kept and what it cost. Judge each of the ten by hand as genuinely useful or not: items 1, 2, 3 and 4 are real descriptions; 5 through 10 are metadata, navigation, or keyword soup. So the pool's useful fraction is q = 4/10 = 0.40, and the kept set's is q′ = 3/3 = 1.00. Precision went to 1 and recall to 3/4 = 0.75 — we lost the true-but-terse product listing at rank 4.

The compute ledger. Suppose the budget is 10 samples seen (scaled to match the pool, as the benchmark does).

Two and a half times the useful compute, at the cost of one distinct useful item. On this toy pool that is clearly a good trade. Push the cut to the top 10% and the ledger flips: useful compute stays at 10.0 while distinct useful items fall to 1, so you would be showing the model one caption ten times. That is the unimodal curve, visible in five rows of arithmetic.

What the filter would have kept, if you asked it differently

One more thing the ten-item table can show. Rank 4 — the accurate but terse product listing — is dropped by an alignment filter and would be kept by several other criteria:

Three of four alternatives keep it. That is the concrete case for why the leaderboard eventually moved past pure score-thresholding: a large fraction of what an alignment filter discards is not junk, it is badly captioned inventory, and discarding is only one of the things you can do with it.

What the leaderboard could not tell you

Every number in this chapter carries conditions. Five worth attaching whenever you quote them.

The three numbers to quote, if you quote anything

Three numbers, three claims, three explicit limits. If you carry the limits with the numbers, you will be a more careful reader of this literature than most of the people citing it.

Chapter 6: Curation by Retrieval

Everything so far has been subtraction. Start with a pool, compute a score, throw rows away. That framing has a hidden assumption: that the pool already contains, somewhere inside it, the dataset you want.

For image-text pairs that assumption is roughly fine. For a different problem it fails completely, and the fix is one of the most elegant uses of an embedding in this whole lesson.

The problem with no captions

Self-supervised visual pretraining wants a large, diverse, balanced image set. No labels needed — that is the point of self-supervision. But this creates an awkward position:

You cannot filter by alignment because there is no text to align with. You cannot filter by class because you have no labels and refuse to buy any. What is left?

The pipeline, with the actual numbers

DINOv2's data pipeline builds LVD-142M — 142 million images — from two ingredients.

Read that last box slowly, because it is the whole idea. The seed supplies distribution; the pool supplies volume. Neither supplies labels. There is no taxonomy anywhere in the pipeline — only a query set and a cosine.

Two retrieval modes, and when each applies

Sample-based retrieval. For each query image, take its N nearest neighbours from the pool, with N typically small — four is the usual setting. Precise, and the retrieved volume scales with the number of queries.

Arithmetic: a query set of 1,000,000 images at N = 4 yields at most 4,000,000 retrieved images before deduplication. In practice fewer are unique, because popular pool images are the nearest neighbour of many queries. Good when the seed is large.

Cluster-based retrieval. When the query set is small, N = 4 gives you almost nothing. So instead: k-means the uncurated pool into a large number of clusters — on the order of 100,000 — then, for each query, take M images from the cluster the query falls into.

Arithmetic: 1.2 billion images over 100,000 clusters gives an average cluster of 12,000 images. A seed of 1,000 images touching, say, 300 distinct clusters and taking M = 1,000 from each yields up to 300,000 images from a 1,000-image seed — an amplification factor of 300.

Why retrieval instead of a classifier

An obvious alternative: train a classifier on the curated seed and use it to score the pool. Keep images it confidently assigns to a seed class. Why is retrieval better?

This is the same argument as open-vocabulary classification, applied one level up. A classifier's vocabulary is welded into a weight matrix; a retriever's vocabulary is whatever you hand it at query time. Applied to models, that observation gives you zero-shot classification. Applied to datasets, it gives you a curation pipeline you can steer with a folder of example images.

The engineering, which is where the real difficulty lives

"Find the nearest neighbours of a million queries in a billion-item pool" is a sentence that hides a hard systems problem. Do the memory arithmetic first, because it explains every design decision that follows.

Storing 1.2 billion embeddings at 1,024 dimensions in float32:

That does not fit in the memory of any reasonable GPU cluster, and streaming it from disk for every query batch is hopeless. The standard fix is product quantisation: split each vector into subvectors, learn a small codebook for each, and store one byte per subvector instead of four bytes per dimension. At 32 bytes per vector:

A 128× compression, and now the index fits across a handful of GPUs with room to spare. Combine that with an inverted-file structure — coarse-cluster the pool, and at query time only scan the few nearest coarse clusters — and the search becomes tractable. The reported infrastructure for this job was on the order of twenty compute nodes with eight GPUs each, running for under two days.

Note what just happened: to build a dataset by retrieval you need a production-grade vector index. The curation problem is a vector-database problem, and the two skill sets are the same skill set.

Deduplicating against your own evaluation

One step in the pipeline is easy to overlook and is genuinely important: near-duplicates of benchmark test images are removed from the pool, using a dedicated copy-detection embedding built for exactly that job.

Why does this matter more for a retrieval pipeline than for a random crawl? Because retrieval actively seeks similarity. If your seed contains ImageNet training images, and the pool contains near-copies of ImageNet test images — and it does, because the web republishes everything — then a nearest-neighbour search is a machine specifically optimised for pulling those copies in. The pipeline's core mechanism is also its contamination mechanism.

What this bought, and the honest framing

The models trained on the retrieved set outperform the same architecture and recipe trained on the raw uncurated pool, and outperform training on the curated seed alone. The gains are largest on fine-grained and dense-prediction tasks — precisely the tasks that need the tail of the distribution, which is precisely what retrieval amplified.

The honest framing is a comparison of three quantities that people conflate:

That middle row is the one that should update your priors. The uncurated pool is roughly eight times larger than the retrieved set and it trains a worse model. More data, worse outcome, same everything else.

Three philosophies, one instrument

Now the three papers can be laid side by side, and the taxonomy is cleaner than it looked in Chapter 0.

Same instrument, three verbs, three failure modes. If you internalise nothing else from this lesson, internalise this table: it is a complete diagnostic checklist for any corpus you are ever handed.

The seed is a prior, and choosing it is the real decision

It is tempting to read "no labels" as "no human judgement." That is wrong, and getting it right changes how you use the method.

The seed collection is the judgement. Someone decided that a general object taxonomy, a landmark collection, and several fine-grained sets together represent "the visual world worth learning." That decision is every bit as opinionated as writing a taxonomy — it is just expressed in examples rather than in category names, and therefore never written down anywhere as a list of commitments.

Retrieval overlap, and why the union is much smaller than the sum

Naively, a million queries retrieving four neighbours each gives four million images. In practice it gives far fewer, and the reason is worth computing because it determines how many queries you need.

Popular pool images are the nearest neighbour of many queries. Model this crudely: suppose the four million retrievals are spread over an effective set of six million "retrievable" images, drawn roughly independently. The expected number of distinct images pulled is:

with P = 6,000,000 retrievable and D = 4,000,000 draws:

Four million retrievals, 2.92 million unique images — a 27% collapse. And the effect worsens as you retrieve more: doubling to eight million draws gives 6M × (1 − e^−1.333) = 6M × 0.7364 = 4.42 million unique, so the second four million draws bought only 1.5 million new images. Retrieval has sharply diminishing returns per query, which is exactly why the pipeline needs a large or well-spread seed rather than a small one queried harder.

The index, in the terms you would actually build it

The memory arithmetic above explains product quantisation. Here is the other half: even compressed, you cannot compare every query to every vector.

An inverted-file index coarse-clusters the pool into nlist partitions and, at query time, scans only the nprobe partitions whose centroids are nearest the query. With a pool of 1.2 billion, nlist = 32,768 partitions hold about 36,600 vectors each. Probing 32 of them scans:

Down from 1.2 billion — a thousandfold reduction — at the cost of missing any true neighbour that fell into an unprobed partition. That is the same one-sided approximation as SemDeDup's clustering step, for the same reason, with the same acceptable failure direction: you miss a few neighbours, you never invent one.

Steering by example: the recipe for a known blind spot

This is the most directly reusable procedure in the lesson. You have a deployed model. It fails on a specific slice. You have a large unlabelled pool.

Step 4 is where this goes wrong in practice. Targeted retrieval is powerful enough that a naive application will happily rebuild your entire training distribution around one incident. Treat the retrieved slice as an ingredient with a mixing weight, not as the new dataset.

All three methods on one corpus, with the numbers flowing through

The three papers are usually read separately. In practice you run them in sequence, and watching a corpus pass through all three is the best summary of the lesson.

Start with a raw crawl of 1,000 million image-text pairs.

Three things to notice in that table, and each is a chapter's thesis compressed to a row.

Stage 3 raises effective diversity while removing items. That is Chapter 2's out-of-distribution result: deduplication is distribution flattening disguised as compression. Removal and narrowing are not the same thing, and conflating them is the most common misreading of curation work.

Stage 4 is the only one that can undo stage 2's narrowing. Every subtractive filter has an anchor and every anchor narrows. Retrieval is the sole operation in the toolkit that can put coverage back, which is why a pipeline of subtractions alone will always drift toward its filters' fixed point.

Stage 5 removes almost nothing and is not optional. 0.2 million items out of 158 million — a rounding error in the dataset and the difference between a measurement and a memory test.

What each stage costs, in one place

The second row is the economic heart of everything in this lesson. One embedding pass, cached, and then five different curation questions are answered with dot products. That is why the practice consolidated around embeddings within about a year of anyone trying it seriously.

Porting retrieval curation to text and to your own domain

Nothing in the method is visual. Strip the images out and the recipe reads: embed a pool, embed a seed, retrieve neighbours, deduplicate, mix. Here is what that becomes in three other settings.

The middle row deserves a sentence of its own, because it inverts how most retrieval corpora get built. The usual approach is to index everything you can and hope coverage follows. Seeding retrieval with real queries makes the corpus a function of demand, and it makes the coverage gap measurable: queries whose nearest pool neighbours are all far away are exactly the topics you have nothing about.

The cold-start problem, and how to get around it

Retrieval curation needs a seed. What if you do not have one?

That last row is worth taking seriously. Retrieval amplifies whatever the seed emphasises, so a seed with three times as many photographs of one category produces a retrieved set with the same imbalance, at a hundred times the volume. Balance the seed, then retrieve. It is a cheap step that is easy to skip and expensive to undo.

Why this closes the loop back to Chapter 1

Recall the three failures and their fixes. Deduplication attacks duplication; alignment filtering attacks noise. Both are subtractive, and subtraction cannot fix skew — removing rows from a Zipf-shaped corpus leaves a Zipf-shaped corpus.

Retrieval is the operation that can put mass back where it is missing. It is the only tool in the lesson that increases the count of anything, and therefore the only one that can raise effective cluster count rather than lower it. Every complete pipeline needs at least one additive stage, and this is it.

A closing contrast: taxonomy versus seed

One comparison holds this chapter together, and it is worth stating in its most general form because it recurs everywhere.

Retrieval curation trades the third row for the fourth. That trade is overwhelmingly worth making at scale — a taxonomy for the visual world is a project, a seed is an afternoon — but the cost is real and it is paid in accountability rather than in accuracy.

Chapter 7: The Loop That Eats Itself

Something has been quietly true in every chapter and we have not named it.

SemDeDup deduplicates LAION using a CLIP embedder. CLIP was trained on a web dataset. LAION itself was assembled by filtering Common Crawl with a CLIP score. DataComp's best baselines score candidate pairs with a CLIP model trained on a different filtered web corpus. DINOv2 retrieves its training set using a self-supervised model, and the resulting DINOv2 is a better retriever for the next round.

This is a closed loop, it has been spinning for several generations of models, and almost nobody analyses it explicitly. It is the reason curation works as well as it does, and it is the source of the field's least-discussed systemic risk.

Why the loop works at all

The first question should be why this is not obviously circular nonsense. If you filter data with a model trained on filtered data, where does new information enter?

Two answers, and both are load-bearing.

Ranking is easier than the task. The filter does not need to be as good as the model you are trying to train. It only needs to rank better than random. A mediocre CLIP model can reliably tell that "a golden retriever puppy sitting in tall grass" matches its photo better than "click here for more" does — a judgement far coarser than the fine-grained classification you actually want. So a weak model can select data that trains a strong one. The loop amplifies because each round asks less of the filter than it delivers in the student.

The pool is new information. Each round draws from a vast unfiltered reservoir the previous model never saw. The filter is not generating data; it is selecting from an external source. As long as the pool contains signal the current model lacks, and the filter is better than chance at finding it, the loop imports information rather than recycling it.

Failure 1: the filter cannot keep what the embedder cannot see

An embedding is a lossy map. Two inputs the encoder maps to the same vector are, to any downstream filter, the same input. There is no threshold, no clustering, no clever aggregation that can recover a distinction the encoder discarded.

Chapter 3's table listed CLIP's weak axes: exact counts, precise spatial arrangement, fine-grained subcategory. Now push that through the loop. A dedup filter built on CLIP will treat "three sheep" and "five sheep" as near-duplicates and delete one. The next model, trained on the survivors, sees fewer counting examples and gets worse at counting. Its embeddings distinguish counts even less. The next round of filtering is more aggressive on the same axis.

This is a ratchet, not an oscillation. Blind spots are self-reinforcing, and the system has no error signal for a capability it never had.

Failure 2: distributional narrowing, quantified

The second failure is measurable, so let us measure it. Use the effective-cluster-count metric from Chapter 1.

Take a toy pool split across five clusters, with occupancies in millions of items:

Entropy, in bits. Each term is p · log₂(1/p):

Now apply a filter whose scorer is better at judging the head than the tail — which is exactly what a filter trained on web data is. Suppose it keeps, in millions: 14, 12, 9, 1.5, 0.2. Total surviving: 36.7.

One round of filtering cost 0.134 bits, taking effective clusters from 3.78 to 3.45 — a factor of 0.913. Small. Now compound it. If each round multiplies effective clusters by the same 0.913:

Five generations of "mild, sensible, individually defensible" filtering and you have lost 37% of your effective diversity. No single round looked wrong. Every round improved the immediate benchmark.

Failure 3: benchmark entanglement

Chapter 5's Finding 4 was a local instance of a general problem. If your filter is anchored on a reference set R, and your evaluation is drawn from something resembling R, then "this filter is better" and "this filter targets my evaluation" are the same statement and you cannot separate them.

This is not misconduct; it is arithmetic. The remedy is procedural rather than technical:

• State the anchor explicitly, always. "Anchored on ImageNet-1k" is a property of a filter, like a hyperparameter.
• Hold out at least one evaluation domain that the anchor demonstrably does not cover.
• Report the anchored and unanchored evaluation separately, the way DataComp reports ImageNet and the 38-task average side by side.

Failure 4: the retrieval-contamination coupling

Covered in Chapter 6 and worth restating as a loop property. Every round of a similarity-driven pipeline is a machine for finding things similar to your reference set. Your evaluation sets are, on the web, republished. So each round pulls evaluation near-copies into training with above-chance efficiency, and the pipeline gets better at doing so as its embedder improves. Decontamination is not a one-time hygiene step; it is a per-round requirement that gets more important as the system improves.

Four mitigations that actually work

The simulation: run the loop and watch the tail

Two experiments to run. First: floor off, aggressiveness 0.8, ten rounds. The fifth cluster goes to essentially zero and the effective count falls by more than half. Second: floor on, same settings, ten rounds. The effective count drops and then stabilises — the floor puts a hard bottom under every cluster. The head clusters look nearly identical in both runs, which is exactly why nobody notices.

What the loop means for your own systems

You may never train a foundation model. You will almost certainly build something with this shape, because the pattern generalises well beyond pretraining.

Every row is the same structure, and every row has the same fix: reserve a random, unfiltered slice; instrument diversity, not just accuracy; and keep the selector's provenance independent of the evaluation.

The actual chain, named

It helps to see that this is not a hypothetical. Trace one lineage:

Every arrow in that table is a real, published step, and the whole chain took about three years. The improvement is genuine and large. The unmeasured quantity is what fell off the edge at each hop.

Failure 5: the selector becomes the target

One more failure, and it is the oldest one in statistics wearing new clothes.

When a measure becomes a target it ceases to be a good measure. In a curation loop, the score is a target for the data rather than for a person, which makes it feel safe. It is not, because the selection process optimises the score exactly as hard as a person would.

Concretely: CLIP score rewards captions in CLIP's own idiom. So a filtered corpus contains disproportionately CLIP-idiom captions. So the next model trained on it is even more confident about CLIP-idiom captions and even less calibrated about everything else. So its scores are even more idiom-selective. The corpus converges toward the fixed point of the scorer — a set of examples the scorer is maximally confident about, which is not the same as a set of examples a learner would benefit most from.

The instrumentation, in code

Chapter 7's mitigations are only worth anything if they are running. Here is the whole dashboard, and it is short:

python — per-round curation health metricsdef curation_health(E_kept, E_pool, labels_kept, labels_pool, k, scores_kept):
    """Log this EVERY time you re-curate. It is the only early warning there is."""
    out = {}

    # 1. Diversity: effective cluster count, kept vs pool
    def eff(lab):
        p = np.bincount(lab, minlength=k) / len(lab)
        p = p[p > 0]
        return 2 ** (-(p * np.log2(p)).sum())
    out["eff_clusters_kept"] = eff(labels_kept)
    out["eff_clusters_pool"] = eff(labels_pool)
    out["diversity_ratio"]  = out["eff_clusters_kept"] / out["eff_clusters_pool"]

    # 2. Where the removals landed. A cluster at 90% is a data-generation bug.
    kept_c = np.bincount(labels_kept, minlength=k)
    pool_c = np.bincount(labels_pool, minlength=k)
    keep_rate = np.divide(kept_c, np.maximum(pool_c, 1))
    out["keep_rate_p05"] = np.percentile(keep_rate[pool_c > 50], 5)   # the starved tail
    out["keep_rate_p95"] = np.percentile(keep_rate[pool_c > 50], 95)

    # 3. Goodhart check: is the kept score distribution TIGHTENING each round?
    out["score_mean"] = float(scores_kept.mean())
    out["score_std"]  = float(scores_kept.std())    # falling round over round = converging
                                                    # on the scorer, not on quality
    # 4. Redundancy left behind
    out["frac_nn_above_095"] = float((nn_cosines(E_kept) > 0.95).mean())
    return out

Four numbers, logged per round, plotted over rounds. Nothing here needs a training run, and the diversity ratio alone would have caught every collapse scenario in this chapter before it cost anything.

Five symptoms that you are in a bad loop

When does the loop stop paying? A stopping rule you can derive

Rounds of curation are not free and not infinitely profitable. Here is a crude model that gives an actual stopping condition, and its crudeness is the point — you can compute it from two numbers you are already logging.

Say the value of a corpus is roughly the useful fraction times the effective diversity:

Each round of filtering does two things. It raises the useful fraction toward 1 by removing some share δ of the remaining junk, and it multiplies diversity by some factor ρ < 1:

The round is worth running only if V goes up:

Set that equal to 1 and solve for the useful fraction at which rounds stop paying:

Now plug in the numbers this chapter already produced. The toy filtering round cost a diversity factor of ρ = 0.913, and suppose each round removes a quarter of the remaining junk, δ = 0.25:

Once your corpus is about 72% useful, another round of that filter destroys more value in diversity than it creates in quality. Keep going and you are actively making the dataset worse, while your benchmark — which is drawn from the head — continues to improve.

Read the bottom-left cell: with an aggressive filter that only slowly improves quality, rounds stop paying once the corpus is barely half useful — which is to say almost immediately. Aggressive filters have to earn their diversity cost with a large quality gain, and most of them do not after the first round.

The one-round special case, which is where most people actually are

Most teams run this loop exactly once: take a pretrained embedder, filter, train. The multi-round analysis then seems irrelevant. It is not, for two reasons.

You are not at round one, the field is. Your pretrained embedder was trained on data curated by a previous generation's filter. You inherit whatever narrowing happened upstream, and you cannot see it, because the only lens you have is the one that produced it.

Deployment is a round. If you retrain periodically on data selected with the previous model — logged interactions, flagged examples, retrieved documents, human labels on what the model surfaced — you are running the loop whether or not you call it that. The instrumentation costs a few lines and the failure it detects is irreversible.

Worked example: five rounds, with and without the floor

The simulation shows this interactively; here it is as arithmetic you can check, because the mechanism matters more than the animation.

Track only the smallest cluster, whose share of the pool starts at 2%. Suppose each round the filter keeps it at a relative rate of 0.60 compared with the average cluster — a plausible penalty for a region the scorer handles poorly. Without a floor, its share is multiplied by roughly 0.60 each round and renormalised, which to a first approximation means:

Without the floor the cluster is heading to zero geometrically and will be gone within a few more rounds. With the floor it converges to a positive limit — solve the fixed point of x = 0.9(0.6x) + 0.1(0.02) and you get x = 0.002/(1 − 0.54) = 0.00435, about 0.44%. The floor does not preserve the cluster's original share. It guarantees the cluster continues to exist, which is the only property that matters, because existence is what a later round can build back from and extinction is not.

Chapter 8: Recipes You Can Run

Everything so far has been about corpora with nine or ten digits. You probably do not have one. You almost certainly have a fine-tuning set, a document collection, or an evaluation suite, and every technique in this lesson applies at that scale with better economics, because at ten thousand items you can compute things web-scale teams have to approximate.

Four recipes, each with the arithmetic worked out.

Recipe A: deduplicate a fine-tuning set

The setting: 40,000 instruction-response pairs assembled from several sources, some overlapping. You suspect redundancy and you want to know how much.

Step 1: choose k. A good default is k ≈ √N, which balances cluster count against cluster size. For N = 40,000:

Step 2: count the work. Pairs per cluster: 200 × 199 / 2 = 19,900. Across 200 clusters: 3,980,000 pairs. At 384 dimensions that is about 4 × 10⁶ × 768 = 3 × 10⁹ floating-point operations. Your laptop does that in under a second. Compare with the global computation: 40,000²/2 = 800 million pairs — also feasible, at 200× the cost and 3.2 GB for the matrix in fp16.

Step 3: pick the threshold from your budget, by bisection. You do not know what ε means for your data. You do know how much you want to remove. So sweep. Here is the shape of a real sweep on a mixed instruction set — your numbers will differ, the shape will not:

The bisection is trivial: evaluate at 0.99 and 0.85, see that 25% lies between them, try the midpoint 0.92, and narrow. Four or five evaluations gets you within a percent of any target, and each evaluation is a threshold comparison over a matrix you already computed.

Step 4: read the per-cluster removal rates. This is the step people skip and it is the most informative output of the whole procedure.

python — the diagnostic that finds your sloprates = []
for c in range(k):
    idx = np.where(labels == c)[0]
    if len(idx) < 5: continue
    removed = (~keep_mask[idx]).mean()
    rates.append((removed, c, len(idx)))

rates.sort(reverse=True)
for r, c, n in rates[:10]:
    print(f"cluster {c}: {n} items, {r:.0%} removed")
    print("  sample:", texts[np.where(labels == c)[0][0]][:140])

If the overall removal rate is 25% but one cluster is at 87%, you have found a template. Someone generated four thousand near-identical examples from one prompt skeleton. That cluster is not a deduplication problem; it is a data-generation problem, and the right fix is upstream: regenerate with more variation, or cap that cluster's contribution deliberately rather than letting a cosine threshold decide.

Step 5: verify against something real. Fine-tune on the deduplicated set with the same number of optimiser steps as the original. If held-out quality holds or improves, you have found free compute. If it drops, your threshold was too aggressive — and note that a drop is informative in itself: it means the "duplicates" carried signal, which usually means your embedder's invariances are wrong for this data.

Recipe B: filter a RAG corpus, where duplicates hurt differently

Near-duplicate chunks in a retrieval corpus cause a failure that has nothing to do with training compute, and it is worse than the training case.

Your retriever returns the top k = 8 chunks. Suppose five of those eight are near-duplicates of one another — the same policy paragraph, copied into five documents. The model receives eight chunks and four distinct facts. The fifth-ranked distinct fact, which might have been the one that answers the question, was pushed to position 9 and never retrieved.

You paid for eight chunks of context, you spent the tokens, and half of it was the same sentence. And the failure is silent: the answer looks confidently sourced, because it is sourced, five times, to one document.

Two fixes, and you want both.

Offline: deduplicate the chunks. Same as Recipe A, but with a much stricter threshold. Use ε around 0.97, because in a knowledge corpus two documents that are 90% similar often differ in the 10% that matters — a version number, an effective date, an exception clause. Deleting one is a correctness bug, not a diversity loss.

At query time: maximal marginal relevance. Select results one at a time, each time trading relevance against novelty:

where S is what you have already selected. Work it by hand with λ = 0.7. Four candidates with similarity to the query:

and pairwise similarities:

Round 1. S is empty, so the penalty term is zero and MMR reduces to plain similarity. Pick A at 0.82. Now S = {A}.

Round 2. Compute for each remaining candidate:

D wins, despite being ranked fourth by raw similarity, because it is the only candidate saying something A did not. S = {A, D}.

Round 3. The penalty is now the max over both selected items:

B, then C. Final order: A, D, B, C instead of A, B, C, D. One reorder, and the context window now contains two genuinely different pieces of evidence in its first two slots instead of two paraphrases.

Recipe C: choose thresholds from budget, never from intuition

Generalise what Recipe A did informally into a procedure.

Step 4 is not optional and it takes ten minutes. Every serious data-curation failure I have seen was visible in a fifty-item sample and invisible in every aggregate.

Recipe D: what order do the filters go in?

A pipeline usually has several stages. The order changes both cost and outcome.

Step 3's parenthetical is a real and slightly counterintuitive effect: filtering for quality first increases the measured duplicate rate, because you have removed the diverse junk and what remains is the well-formed, frequently-republished material. If you dedup first and quality-filter second, you will under-remove duplicates and then wonder why your redundancy metrics look fine while your model memorises.

The corpus cheat sheet

Recipe A, as one runnable script

python — deduplicate a fine-tuning set, end to endimport numpy as np, json

texts = [ex["prompt"] + "\n" + ex["response"] for ex in data]   # 40,000
E = embed(texts)                                    # (40000, 384)
E = E / np.linalg.norm(E, axis=1, keepdims=True)

# 1. LOOK before deciding. Nearest-neighbour cosine histogram.
S = E @ E.T
np.fill_diagonal(S, -1)
nn = S.max(axis=1)
for q in [50, 75, 90, 95, 99]:
    print(f"p{q} nearest-neighbour cosine: {np.percentile(nn, q):.4f}")

# 2. Bisect eps to a REMOVAL TARGET derived from your compute budget.
def removed_at(eps):
    keep = np.ones(len(E), dtype=bool)
    for i in range(len(E)):                       # greedy: earlier survivors win
        if not keep[i]: continue
        dup = np.where((S[i] > eps) & keep)[0]
        keep[dup[dup > i]] = False
    return keep

lo, hi, target = 0.80, 0.999, 0.25
for _ in range(8):                                 # 8 steps is plenty
    mid  = (lo + hi) / 2
    frac = 1 - removed_at(mid).mean()
    if frac > target: lo = mid
    else:            hi = mid
eps  = (lo + hi) / 2
keep = removed_at(eps)
print(f"eps={eps:.4f}  removed={1-keep.mean():.1%}")

# 3. READ FIFTY REMOVED ITEMS. Non-negotiable.
for i in np.where(~keep)[0][:50]:
    print("---", texts[i][:160].replace("\n", " | "))

json.dump([d for d, k in zip(data, keep) if k], open("deduped.json", "w"))

Note the bisection direction: raising ε removes less, so when the removed fraction is above target you move the lower bound up. Getting that backwards produces a script that confidently converges on the wrong end, and it is the single most common bug in this procedure.

Recipe E: decontaminate an evaluation set

Different job, different tool, and it belongs in every pipeline that touches web data.

The question is not "do these mean the same thing?" but "is one of these derived from the other?" Semantic similarity is the wrong instrument — two different questions about the same topic are semantically close and are not contamination. Use a copy-detection descriptor, or for text, exact and near-exact substring matching.

Worked: an evaluation set of 50,000 items, a training candidate set of 40,000,000. You find 1,900 training items above threshold. Removing them costs 0.005% of your training data. Not removing them puts an unknown fraction of your evaluation inside your training set. There is no version of that trade where you keep them.

Three ways this goes wrong in practice

What to log, every time

Recipe F: triage a corpus somebody handed you, in thirty minutes

The most common real situation: you inherit a dataset, nobody remembers how it was built, and you need to know whether to trust it before you spend a training run on it.

Thirty minutes and one GPU produce a paragraph you can put in a document: how many rows, how many distinct, how much near-duplication, how many effective clusters, and what the five biggest clusters actually are. That paragraph is worth more than most dataset cards.

The thresholds table, with the reasoning attached

Numbers without reasons do not transfer, so here is why each recommendation in the corpus cheat sheet is where it is.

The mixing question nobody asks until it is too late

Curation produces a set. Training consumes a stream, and the mapping between them is a policy you are choosing whether or not you notice.

The honest state of practice: uniform is the default, balanced-by-cluster is the cheapest improvement, and explicit source weights are what you actually want but require provenance you probably discarded. If you are building a pipeline from scratch, carry a source identifier on every row from the very first stage. It costs a column and it is the thing you will most wish you had.

Chapter 9: Legacy & Cheat Sheet

Three papers, one instrument, and a change in what the field considers a contribution. Here is what happened next and what is still broken.

One table for the whole lesson

Every number in this lesson, in one table

What the three papers actually contributed

What came next

And the sibling line of work on the deduplication side: combining semantic deduplication with diversity-aware selection for language-model pretraining, which is the direct descendant of SemDeDup and confirms the same finding in a different modality.

What is still broken

Connections across the site

The sentence to leave with

Strip everything away and one claim remains, and it is falsifiable, which is what makes it worth holding.

The first half of that sentence is what the three papers established with controlled experiments. The second half is what makes it actionable rather than merely true: the measurement is cheap, it is available before you spend anything, and nothing about it requires a frontier lab.

The uncomfortable corollary is that most teams do not make the measurement, which means most teams are choosing their model's ceiling by accident. That is not a criticism of anyone — the instrument is two years old and the habit has not spread yet. It is an opportunity, and it expires as the habit becomes normal.

Questions to take back to the papers

Reading a paper with a question is a different activity from reading it to find out what it says. Here are the questions this lesson leaves open, matched to where the answers live.

The last two rows are not rhetorical. They are the two measurements this field most obviously needs and has not made, and both are within reach of a small group with a modest cluster and the patience to build the instrument first.

Objections, and what they are worth

A lesson should be able to state the strongest case against itself. Five objections, each with the part that is right and the part that is not.

Five exercises, in increasing order of ambition

Exercise 4 is the one that changes how you work, and it is the one people skip because it produces no artifact except a number. That number is the whole point: it is the first time you will have measured a data decision instead of believing one.

The last thing to say about all of this

The three papers in this lesson share an unusual property. None of them introduced a new architecture, a new loss, or a new optimiser. SemDeDup is k-means and a threshold. The benchmark is a pool and a frozen script. Retrieval curation is a nearest-neighbour search. Every component was decades old.

What they did was notice that a variable everyone had frozen was worth unfreezing, and then build the instruments to measure it. That is a different kind of contribution from the one the field was set up to reward, and it turned out to be worth more than most of the alternatives available at the time.

The three-year arc, in order

Seven steps, each one enabled by the last, and the whole arc runs on the same primitive: comparing two vectors.

If you remember one paragraph

What this changes about how you spend a week

Every row on the right takes less than a day and costs no GPU-hours worth mentioning. That asymmetry — a day of measurement against a week of training — is the practical reason this material pays for itself immediately, whatever scale you work at.

The five ideas to keep

How to read the data section of a paper

A practical skill this lesson should leave you with. Data sections are short, passive, and load-bearing. Here is what to look for.

What changed about what counts as a contribution

The deepest effect of these three papers is not any of their techniques. It is that they made "we built a better dataset" into a claim that can be checked.

Before, the sentence was unfalsifiable, so it was rarely made and never argued about. After, it comes with a protocol: freeze the recipe, publish the mask, report the suite. Data work moved from infrastructure to research, and the practical consequence is that curation knowledge now accumulates the way architecture knowledge did between 2012 and 2018 — each result standing on the last, on a shared instrument.

Open problems worth working on

Notice that four of the five are measurement problems rather than algorithm problems. That is the shape of a field right after it acquires an instrument: the next bottleneck is always what the instrument does not see.

Build it yourself — the weekend version

References worth reading next

1. Abbas, A., Tirumala, K., Simig, D., Ganguli, S., Morcos, A. S. "SemDeDup: Data-efficient learning at web-scale through semantic deduplication," 2023 — arXiv:2303.09540. The core method of Chapter 2.
2. Gadre, S. Y. et al. "DataComp: In search of the next generation of multimodal datasets," NeurIPS 2023 Datasets and Benchmarks — arXiv:2304.14108. The benchmark of Chapters 4 and 5.
3. Oquab, M. et al. "DINOv2: Learning Robust Visual Features without Supervision," 2023 — arXiv:2304.07193. Read the data-processing section for Chapter 6.
4. Sorscher, B., Geirhos, R., Shekhar, S., Ganguli, S., Morcos, A. S. "Beyond neural scaling laws: beating power law scaling via data pruning," NeurIPS 2022 — arXiv:2206.14486. The theoretical licence for the whole enterprise.
5. Lee, K. et al. "Deduplicating Training Data Makes Language Models Better," ACL 2022 — arXiv:2107.06499. Exact and MinHash deduplication for text, and the memorisation result.
6. Schuhmann, C. et al. "LAION-400M," 2021 — arXiv:2111.02114; "LAION-5B," 2022 — arXiv:2210.08402. Where CLIP-score filtering became standard practice.
7. Fang, A. et al. "Data Filtering Networks," 2023 — arXiv:2309.17425. Training a network whose only job is filtering, and the decoupling of filter quality from model quality.
8. Xu, H. et al. "Demystifying CLIP Data," 2023 — arXiv:2309.16671. Metadata balancing as an alternative to score-based filtering — the Chapter 7 escape hatch.
9. Maini, P. et al. "T-MARS: Improving Visual Representations by Circumventing Text Feature Learning," 2023 — arXiv:2307.03132. The rendered-text shortcut in CLIP score.
10. Nguyen, T. et al. "Improving Multimodal Datasets with Image Captioning," 2023 — arXiv:2307.10350. Rewriting data instead of discarding it.
11. Tirumala, K. et al. "D4: Improving LLM Pretraining via Document De-Duplication and Diversification," 2023 — arXiv:2308.12284. SemDeDup's direct language-model descendant.
12. Li, J. et al. "DataComp-LM: In search of the next generation of training sets for language models," 2024 — arXiv:2406.11794. The DataComp design ported to text.
13. Pizzi, E. et al. "A Self-Supervised Descriptor for Image Copy Detection," CVPR 2022 — arXiv:2202.10261. The copy-detection embedder used for decontamination.
14. Johnson, J., Douze, M., Jégou, H. "Billion-scale similarity search with GPUs," 2017 — arXiv:1702.08734. The index that makes all of this runnable.
15. Radford, A. et al. "Learning Transferable Visual Models From Natural Language Supervision" (CLIP), 2021 — arXiv:2103.00020. The embedder underneath nearly every filter here.