A web scrape of 12.8 billion image-text pairs is mostly noise. Choose which pairs to keep, and the same model reaches 79.2% on ImageNet, 3.7 points above OpenAI’s CLIP.
Learn how researchers filter the web into a training set, and how they build tests that a vision model cannot pass by reading alone.
Pick a rule, then drag Keep the top from loose to strict. Watch how many kept pairs are good, and how many rare concepts vanish. Then we build, piece by piece, the scores, the filters, the controlled test bed and the benchmarks that make this measurable.
You need a rough idea of what a neural network is and a little Python. We teach embeddings, similarity scores and benchmark audits from zero.
Built from eight papers: DataComp, MetaCLIP, Data Filtering Networks, Molmo and PixMo, MMStar, Eyes Wide Shut, BLINK and Cambrian-1, re-taught in our own words and devices.
Illustrative toy pool: 160 made-up pairs, 20 of them about a rare concept, about 37% (59 of 160) truly good. Scores are drawn from overlapping bell curves, and the CLIP-style score is set slightly lower for rare concepts to show the bias we measure in Chapter 9. The rules and the counting are exact; the data are not a real crawl. Real numbers come in Chapters 2 to 8.
Chapter 0
Read eight raw pairs from the web, sort them yourself, and see why every vision model starts with a filter
Imagine you want to teach a computer what the world looks like. You have no photographers and no budget for labels. But the web already holds billions of pictures, and most of them sit next to some text. So you write a program that visits web pages and saves every picture together with the words attached to it.
Such a program is a web crawler, and the file it produces is a web scrape. Each saved item is an image-text pair: one picture and one string of words that is supposed to describe it. The words usually come from the alt-text, a short note that a web page author can attach to a picture for screen readers and search engines.
Here is the catch. Nobody wrote that note to teach a model. The author wrote it for a search engine, or for a layout tool, or not at all. So before you train anything, you must ask a plain question: how many of these pairs are any good?
The device below shows eight pairs. Each one is a small record, the kind a data pipeline stores, plus a line that says what the picture shows. Decide whether the words describe the picture well enough to train on. Press Keep or Drop for each.
Read the record. Ask: if a model saw only this picture and these words, would it learn something true about the picture? Then choose.
Illustrative: eight made-up records written to show the common kinds of web noise. The mix is typical in kind but is not a measured rate from any real crawl.
You may keep three or four and feel unsure about a few. That unsureness is the point. Two people will disagree on a blurry case, and a program has to make that call billions of times.
Look at what went wrong in the dropped pairs. They fall into a few families, and each family needs its own cure.
Boilerplate is text copied from the page frame, such as “Home | About us | Contact”. It says nothing about the picture. Filenames such as “IMG_4021.jpg” are the same kind of problem: words that exist only because a camera wrote them. Advertising text sells something and does not describe it. Article text describes the story around the picture, not the picture. Tiny images, such as a 16 by 16 icon, have too few pixels to show anything.
There is also a quieter problem. A pair can be correct and still useless, because it is the ten thousandth photo of a plain white mug with the caption “mug”. We call that a near-duplicate, and Chapter 3 shows how to remove it.
You could sort pairs by hand, as you just did. Now do the arithmetic. The testbed we study in Chapter 4, called DataComp, starts from a pool of 12.8 billion pairs taken from the Common Crawl web archive. If a person reviews one pair every five seconds, 12.8 billion pairs take about 2,000 person-years of nonstop work. (This is our arithmetic: 12.8 billion times five seconds is 64 billion seconds, and a year has about 31.6 million seconds.) No team does that.
So filtering must be automatic. A program must score each pair, and a rule must keep the good ones. The whole field of vision data curation is about choosing that score and that rule well. The hero at the top of this page is a toy version of exactly that choice.
Training data is only half the story. After you train a model, you test it on a benchmark, a fixed list of questions with known answers. A benchmark is also built from raw material, and it can also be noisy. A question might be answerable without looking at the picture at all. A test image might have leaked into the training set. If so, a high score tells you nothing.
So this lesson has two halves that mirror each other. First we clean what the model learns from. Then we clean what the model is tested on. The same habit runs through both: measure, do not guess.
A filter has two jobs that pull against each other. It must throw away junk. It must also keep rare, valuable things, such as a photo of an unusual bird that few pages mention. A strict filter does the first job well and the second badly. The hero shows this: drag it strict, and the share of good pairs rises while the rare lane empties. We will measure both sides in every chapter.
Chapter 1
Turn a picture and its words into points on one map, and read their distance as a quality score
You sit in a room with a million photos and a million sentences. A friend asks: which sentence belongs to this photo? You cannot read a million sentences. But suppose each photo and each sentence had a spot on one big floor map, and matching items stood close together. Then you would walk to the photo and look at the nearest sentence.
That map is the idea behind the CLIP score, the most common first filter for image-text pairs. The rest of this chapter builds the map, the distance and the cut, with real numbers.
A vector is a list of numbers. Think of it as the coordinates of a point, or as an arrow from the centre of the map to that point. A map with two numbers is a flat floor. Real models use hundreds of numbers, for example 512, but the geometry works the same way.
An embedding is the vector a neural network gives to a picture or to a sentence. An image encoder reads pixels and returns one vector. A text encoder reads words and returns one vector of the same length. The data flow is short:
shapesimages (B, 3, 224, 224) -> image encoder -> (B, 512)
captions (B, 77 tokens) -> text encoder -> (B, 512)
both are scaled to length 1, so each is a point on a sphere
The two encoders are trained together with a contrastive loss. In plain words: take a batch of true pairs. Pull each picture toward its own caption. Push it away from every other caption in the batch. After training on hundreds of millions of pairs, true pairs end up close and unrelated pairs end up far. Our CLIP lesson builds this training step in full.
How do we measure closeness between two arrows? We use the angle between them. Two arrows pointing the same way have angle 0. Two arrows at right angles are unrelated. The number we use is the cosine similarity: the cosine of that angle. It is 1 for the same direction, 0 for right angles, and negative for opposite directions.
To compute it, multiply the two vectors position by position and add (the dot product), then divide by the two lengths. Here is a worked example with three numbers per vector. All numbers are made up for teaching.
Let the image vector be a = (0.8, 0.5, 0.1). Let the caption “red bicycle” be b = (0.7, 0.6, 0.2). The dot product is 0.8×0.7 + 0.5×0.6 + 0.1×0.2 = 0.56 + 0.30 + 0.02 = 0.88. The lengths are √0.90 = 0.949 and √0.89 = 0.943. So the score is 0.88 / (0.949 × 0.943) = 0.983. Now take the caption “buy now free shipping”, c = (0.1, −0.2, 0.9). Its dot product with a is 0.08 − 0.10 + 0.09 = 0.07, its length is 0.927, and the score is 0.07 / (0.949 × 0.927) = 0.080. The good caption scores about twelve times higher.
Drag the caption around the map below. The image sits at a fixed spot. The gauge on the right shows the score, and the dashed line is your cut.
The blue arrow is the picture. The warm arrow is the caption. Slide the angle between them, or tap a preset caption. Raise the cut to be stricter.
Illustrative: a two-number map and made-up preset angles. The cosine is exact for the angle you set. Real encoders use hundreds of numbers, and real cuts are set by looking at kept samples, not by a rule of thumb.
A researcher writes the score as a few lines. The pair is kept when the score clears the cut. Notice that the threshold is a single number for the whole dataset.
pythonimport torch, open_clip model, _, prep = open_clip.create_model_and_transforms("ViT-B-32", pretrained="...") tok = open_clip.get_tokenizer("ViT-B-32") def clip_score(images, captions): # images: (B,3,224,224) with torch.no_grad(): a = model.encode_image(images) # (B, 512) b = model.encode_text(tok(captions)) # (B, 512) a = a / a.norm(dim=-1, keepdim=True) b = b / b.norm(dim=-1, keepdim=True) return (a * b).sum(-1) # (B,) cosine per pair keep = clip_score(imgs, caps) > 0.30 # one cut for everything
Symptom. You filter a crawl with one cut. The result looks clean. But when you search it for an unusual bird or a regional dish, you find almost nothing.
Measurement. The score is a model’s opinion, and the model only has a sharp map for things it saw often. For rare things its picture and caption arrows sit farther apart even when the pair is perfect. So count survivors per concept before and after the cut. In the hero, the rare lane lost most of its pairs while the common lane stayed healthy.
Fix. There are two. Train a better judge (Chapter 2). Or stop using one global cut and balance the concepts (Chapter 3).
Chapter 2
Compare a plain CLIP-score cut with a filter trained on clean data, and see why a good judge is not the same as a strong classifier
A teacher marks a pile of 160 essays. She has time to keep only the best 48 for a class booklet. How does she choose? She can use a quick rule, such as “the longest essays first”. Or she can use a trained colleague who has read thousands of good essays. The colleague takes longer to find, but the booklet will be better.
Image-text filtering faces the same choice. Chapter 1 gave us the quick rule: keep pairs whose CLIP score is high. This chapter adds the trained colleague, and then compares the two on one pool.
The procedure has four steps. Score every pair. Sort the scores from high to low. Keep the top share, say 30%. Train on what is left. Each pair now carries a small extra field in its record:
json{"url": "https://.../bike.jpg", "caption": "red bicycle leaning on brick wall",
"clip_score": 0.31, "keep": true}
{"url": "https://.../blender.jpg", "caption": "BUY NOW free shipping 20% off",
"clip_score": 0.09, "keep": false}
This is cheap and it works well enough to be a standard first step. Its weakness is that the judge is a general-purpose model. It was trained to match pictures with words, not to tell a useful training pair from a useless one.
A data filtering network, or DFN, is a model trained for one job: pick good training data from a huge uncurated pool. The paper behind it, Data Filtering Networks, starts from the step every lab already does, which is filtering a web-scale pool down to a training set with hand-made rules. It asks whether a learned model can do that step better.
The recipe has five steps:
The paper reports one result that surprises people. In its words, the quality of a network for filtering is distinct from its performance on downstream tasks. A model that scores well on ImageNet can produce a worse training set than a weaker model trained on a small amount of high-quality data. Think of chess. The world’s best player is not always the best coach, because coaching needs a different skill: knowing which positions teach the most.
The headline number from the same paper: a ViT-H model trained on the filtered dataset DFN-5B reaches 84.4% zero-shot accuracy on ImageNet. (Zero-shot means the model answers without any ImageNet training pictures.) The authors also release DFN-2B, a dataset of 2 billion examples, and report that their model outperforms models trained on LAION-2B, DataComp-1B, or OpenAI’s WIT data.
Now see the difference on the hero’s own toy pool. Both judges rank the same 160 pairs. As you keep a smaller and smaller top share, the question is how many of the kept pairs are truly good. Drag the share and switch the measure.
Each line follows one rule as you keep more of the pool. Move the vertical marker with the slider. Choose what to measure.
Illustrative toy pool (the same 160 pairs as the hero). The shape is the lesson: a learned judge tends to stay clean longer as you keep more. The numbers belong to the toy, not to any paper. The balanced rule cuts each concept group separately, which Chapter 3 explains.
On this pool, keeping the top 30% means 48 pairs. The CLIP-score cut leaves 38 good ones, which is 79.2%. The learned filter leaves 44, which is 91.7%. The gap is real but it is not magic: the learned judge was only given a better definition of “good”.
Look at the left end of the lines in the device. At a 10% keep share the learned filter keeps 16 pairs and all 16 are good. The CLIP cut keeps 16 and 14 are good. As the share grows, both lines fall, because the judge must now accept pairs it likes less. At 100% both reach the pool’s base rate of 59 good pairs in 160, which is 36.9%. Two facts follow. First, every filter trades quantity for quality: the cleaner set is also the smaller set. Second, the better judge buys you a higher line, which means you can keep more pairs at the same cleanliness. More pairs at the same quality is the real prize, because a model learns more from a bigger clean set.
Symptom. Your filtered set is clean, yet the model trained on it is weak on unusual objects.
Measurement. Switch the device to the rare-concept measure. At a 30% keep share, the CLIP cut keeps 1 of the 20 rare pairs and the learned filter keeps 4. A clean set can still be a narrow set, because any judge is happiest with the common cases it knows best.
Fix. Add a step that protects the tail. Chapter 3 shows how a cap on common concepts does that.
Chapter 3
Cap the crowded concepts, keep the rare ones, and drop the pairs that say the same thing twice
Picture a school choir with 500 children. Fifty thousand parents want a ticket to the concert, and most of them come from one street. If you hand tickets out at random, the audience is almost all one street. If you give each street at most ten tickets, the hall fills with many different faces.
A web scrape has the same shape. A few concepts, such as “person” or “dog”, fill most of the pool. Thousands of others appear a handful of times. This crowding is called a long tail: a few very tall bars on the left, and a long thin strip of short bars on the right.
A model learns from what it sees most. If 89% of the pairs are about three concepts, the model spends almost all of its effort on those three. A rare concept appears so seldom that the model barely learns it. And remember Chapter 1: a global CLIP-score cut tends to drop rare pairs first, so a filter alone makes the crowding worse.
The paper Demystifying CLIP Data, which introduced MetaCLIP, attacks this directly. It starts from two inputs: a raw data pool, and a list of metadata entries, which are concept names derived from the concepts CLIP itself used. The method then yields a balanced subset over the metadata distribution.
In plain words there are three moves. First, match each caption to the metadata entries it mentions. Second, count how many pairs each entry has. Third, set a cap t. For an entry with more than t pairs, keep a random sample of size t. For an entry with fewer, keep all of them. The tall bars are cut down and the tail is untouched.
Here is a small worked case. The counts below are made up. Take ten concepts: person 50,000, dog 20,000, car 12,000, pizza 6,000, bicycle 3,000, accordion 800, kayak 400, heron 120, astrolabe 25, theremin 8. They total 92,353 pairs. The three tallest concepts hold 82,000 of them, which is 88.8%. The five rarest hold only 1,353 pairs, which is 1.5%.
Now set t = 1,000. The five crowded concepts keep 1,000 each, which is 5,000. The five rare ones keep everything, 1,353. The new total is 6,353. The top three now hold 3,000 of those, or 47.2%, and the rare five hold 21.3%. The set is far smaller, but each rare concept now has the same chance as before, and a far larger share of the whole. Play with the cap below.
Blue bars: pairs per concept in the raw pool. Green bars: pairs kept after the cap. The scale is logarithmic, so each gridline is ten times the one before. Slide the cap from loose to tight.
Illustrative: ten made-up concept counts, chosen to show a long tail. Real metadata lists hold far more entries, and the real method picks its cap with care; see the MetaCLIP paper. The capping arithmetic is exact.
What did the paper get from this? On a CommonCrawl pool, MetaCLIP curated 400 million image-text pairs, and a ViT-B model trained on them reaches 70.8% zero-shot accuracy on ImageNet, against 68.3% for CLIP. Scaled to 1 billion pairs at the same training budget, it reaches 72.4%, and a ViT-H model reaches 80.5%. The curation code and the metadata distribution are public, which is part of the point: the recipe is open.
Balancing treats concepts. Deduplication treats repeats. A viral image appears on thousands of pages, each with slightly different alt-text. Training on all of them spends your budget re-learning one picture. Worse, if a test image is among the copies, the test is no longer fair (Chapter 9).
There are two levels. Exact deduplication hashes the image bytes and drops repeated hashes. It is cheap and catches only perfect copies. Near-duplicate removal catches resized, cropped or recompressed copies by comparing embeddings: if two images have a cosine similarity above a threshold, keep one. A worked check: two embeddings with cosine 0.97 and a threshold of 0.95 are near-duplicates, so one is dropped.
pythonimport hashlib, numpy as np seen, kept = set(), [] for rec in shard: # exact copies first h = hashlib.sha256(rec["image_bytes"]).hexdigest() if h not in seen: seen.add(h); kept.append(rec) E = np.stack([r["emb"] for r in kept]) # (N, 512), unit length drop = np.zeros(len(kept), bool) for i in range(len(kept)): # near copies: greedy if drop[i]: continue sims = E[i+1:] @ E[i] # (N-i-1,) cosines drop[i+1:] |= sims > 0.95 # mark later copies
The loop is quadratic, so web-scale systems group embeddings first and compare only inside a group. Our CS336 data lesson covers the engineering of filtering and deduplication at that scale.
Symptom. After capping, the model improves on rare objects but loses a little on everyday ones.
Measurement. Evaluate on two slices: common concepts and rare concepts. A cap that is too tight starves the head. A cap that is too loose changes nothing. Sweep the cap, as the device does, and plot both slices.
Fix. Pick the cap on a held-out set that matches how the model will be used. And dedupe before you balance, so the cap counts distinct pictures and not copies.
Chapter 4
Fix the model and the compute, change only the data, and read the winner off a leaderboard
Two bakers each claim to have the best flour. One bakes in a wood oven for an hour. The other uses an electric oven for forty minutes with a new recipe. The second loaf tastes better. Was it the flour? You cannot tell, because five things changed at once.
For years, dataset papers looked like this. One team trained one model on its new dataset. Another team trained a different model on another dataset. The scores differed, but so did the code, the model size and the training time. Nobody could say which dataset was better.
A controlled experiment changes one thing and holds everything else still. DataComp applies this to image-text data. The organizers publish a fixed candidate pool of 12.8 billion image-text pairs from Common Crawl. They also publish fixed training code, a standard CLIP recipe. Every team must train with that code. Every trained model is tested on the same 38 downstream test sets.
The only thing a team may change is the data. It can design a filtering technique that picks a subset of the pool, or curate new data sources. Then it trains with the fixed code and submits a score. The benchmark runs at several scales, spanning four orders of magnitude of compute, so that cheap experiments on small scales can guide choices for expensive large ones.
| Held fixed | Allowed to vary |
|---|---|
| Candidate pool: 12.8 billion pairs | Which pairs you keep (filtering) or add (new sources) |
| Training code: a standard CLIP recipe | Nothing in the model or the optimizer |
| Compute scale chosen up front | How many pairs that budget is spent on |
| Evaluation: 38 downstream test sets | Nothing |
The paper’s best dataset, DataComp-1B, trains a CLIP ViT-L/14 model from scratch to 79.2% zero-shot accuracy on ImageNet. The same procedure and compute with OpenAI’s data gives a model that is 3.7 percentage points lower. So OpenAI’s ViT-L/14 sits at 79.2 − 3.7 = 75.5% on that test (our subtraction from the abstract). Because the model, code and compute are matched, the gap is attributed to the data. Nothing else changed.
This is why the pool is a useful thing to study. It turned “which dataset is better” from an argument into a measurement. The several compute scales also let a team check whether a filter that wins when training is cheap still wins when training is expensive.
A fixed compute budget means the model sees a fixed number of samples, no matter how many pairs you keep. Keep too few and the model sees the same pairs again and again. Keep too many and the junk comes back. The best share is somewhere in between. Try the toy version: choose a rule, then slide the keep share and watch the toy score.
Model, code and compute are fixed (shown as chips). Only the data rule and the keep share change. The line shows a toy score; the dot is your run.
Illustrative toy model, not DataComp’s numbers. Toy score = 100 × (share of kept pairs that are good) × (1 − e−6 × share kept); the second factor stands for the model seeing too few distinct pairs when the budget is fixed. The good-share values come from the 160-pair pool of the hero. DataComp’s real results are the two numbers in the text above.
Two lessons hold for any fixed-budget test, and the toy shows both. First, a moderate keep share can beat both extremes. Second, the best rule at one scale is not guaranteed to be best at another, so run the cheap scales before you pay for the big one.
It helps to see the data flow. A team starts with the pool, which is a long list of pairs. Its filtering method scores each pair and decides which to keep. For teaching, think of the output as a list of the kept pairs’ ids, for example 140 million ids out of 12.8 billion. The fixed training code then reads only those pairs, trains a CLIP model, and runs the 38 test sets. A single number, such as ImageNet accuracy, or an average across the sets, goes to the leaderboard. Because the code is identical for everyone, a team can change its filter overnight and compare against yesterday’s run with no other change to explain.
Symptom. Your filter tops the small-scale leaderboard, then falls behind at the large scale.
Measurement. Run the same filter at two or more scales and plot the score against compute. If the lines cross, the filter’s advantage depends on scale.
Fix. Treat small-scale wins as hypotheses, not conclusions. Check a rule at the largest scale you can afford before you bet a training run on it.
Chapter 5
Compare a scraped alt-text with a long human description, and count how many questions each one can answer
Show a friend a street photo and ask for a caption. One friend says, “bicycle”. Another says, “A red bicycle leans on a brick wall. A small dog sits to the right, next to a blue mailbox. A sign above reads OPEN.” Both are true. Only one lets you check whether a model really understood the picture.
That gap is the subject of this chapter. Filtering can only choose among the captions the web already has. If the best available caption says “bicycle”, the model can learn no more than that. Another path is to write new captions.
Here is a plain way to compare captions. Write a list of questions about a picture whose answers are visible in it. For each caption, count how many of those questions it answers. Call this its question coverage. A model trained to produce or use a caption learns, at best, the facts that the caption contains. So higher coverage means richer supervision.
Try it on a toy street scene. Pick a caption and see which questions it covers. The scene is simple, so you can check each answer with your own eyes.
The scene shows six things. Choose a caption. Rings mark the things it mentions. The list marks the questions it can answer.
Illustrative: one made-up scene, three made-up captions and eight made-up questions. Whether a caption answers a question was decided by hand for this toy. The idea, counting checkable facts, is the one to keep.
Alt-text covers one question out of eight. The one-sentence caption covers three. The dense caption covers all eight. Nothing about the scene changed. Only the label did.
The Molmo and PixMo paper builds a family of open vision-language models and a set of datasets called PixMo. The abstract lists three of its datasets:
All three were, in the paper’s words, collected without the use of external vision-language models. That sentence matters. The abstract says the strongest open-weight models rely heavily on synthetic data from proprietary vision-language models. In other words, they learn from captions another model wrote. PixMo is an attempt to break that chain with fresh human work.
A pointing record looks like this (the field names here are ours, for teaching):
json{"image": "street_0412.jpg", "prompt": "Point to the dog.", "points": [[0.78, 0.64]]} // x, y as fractions of width and height
Why does a pointing answer help? Because it can be checked by a program. A caption is a paragraph that a person has to read. A point is two numbers. You can score it by distance, and a model can learn to ground its words in pixels.
The Molmo model family includes a 72B model (72 billion parameters). The abstract states that it outperforms larger proprietary models, including Claude 3.5 Sonnet and Gemini 1.5 Pro and Flash, and is second only to GPT-4o, judged on both academic benchmarks and a large human evaluation. The abstract does not give dataset sizes, so we do not quote any here; read the paper for them.
Read this as an existence proof, not a ranking. The abstract does not compare datasets on equal terms, so we do not say PixMo beats any other dataset. It shows that a strong open model can be built on data collected without another model’s output.
A dense caption is not just a long caption. It is a caption that names things, their attributes and their places. Look at the dense caption in the device. It names six things. It gives colours (red, blue), a material (brick), positions (to the right, above, far left) and the words printed on the sign. Each of those is a fact a model can check itself against. A paragraph of vague praise, such as “a lovely scene full of charm”, is long and covers zero questions. So when you review human captions, do not count words. Count facts, and spot-check them against the image.
Symptom. A model trained mostly on short scraped captions gives vague answers. Asked “what does the sign say?”, it says “a sign”.
Measurement. Build a coverage list for a sample of 100 training images. Count the questions the captions can answer, as the device does. A low average tells you the label is the limit, not the model.
Fix. Add dense captions where it counts, usually in the pre-training stage that teaches the model to describe, and a question-answer set in fine-tuning. Human writing is slower and costs money, so spend it where coverage is lowest.
Chapter 6
Hide the image from the model, count what it still gets right, and learn why that number should be small
A teacher gives a class a quiz titled “Reading Charts”. A clever student covers the chart with a hand and still answers most questions, because the questions are common knowledge. “Which planet has rings?” does not need a chart. The quiz says it tests chart reading. It really tests general knowledge.
Many vision benchmarks have this flaw. The next two chapters show how researchers found it, and how they built tests that avoid it.
A vision benchmark is a fixed list of items. Each item has an image, a question about it, a few answer choices, and one correct choice. You run a model on every item and report the share it gets right, its accuracy. People then compare models by that single number. So the number must mean what it claims to mean. If the questions can be answered without looking, the number measures something else.
The MMStar paper, Are We on the Right Way for Evaluating Large Vision-Language Models?, names two problems.
The first: visual content is unnecessary. The answer can be worked out from the question text, the answer choices, or a language model’s world knowledge. The paper gives a striking case: Gemini Pro scores 42.9% on the MMMU benchmark without any visual input. Across six benchmarks it beats a random-choice baseline by more than 24% on average, again with no image.
The second: unintentional data leakage. Language models and vision-language models sometimes answer questions that truly need the image, without seeing it. The authors suggest the samples were memorized during training. Their example: the model Sphinx-X-MoE gets 43.6% on MMMU without accessing images, which is 17.9 points above its own language-model backbone. If a bigger multimodal model beats its text backbone while blind, something in its training touched the test.
Here is the simplest honest check. Run the benchmark twice. First normally, with images. Then blind: remove the image, or replace it with a blank, and ask the same questions. Call the two scores the sighted and blind accuracy. Also compute the random-chance score: with four answer choices, chance is 1 in 4, which is 25%.
Read the three numbers together. If blind accuracy is near chance, the questions need the picture. If blind accuracy is far above chance, many questions can be answered by text alone, and the sighted score is inflated. The difference between sighted and blind accuracy is the gain the image brings. The MMStar paper turns this into two metrics, one for leakage and one for the actual performance gain from multimodal training. We keep the plain version.
Now work through one toy audit. A benchmark has 40 questions with four choices each. After testing we know three things about every question.
Blind accuracy is (5 + 13 + 6) / 40 = 24 / 40 = 60%. Sighted accuracy is (12 + 14 + 8) / 40 = 34 / 40 = 85%. Chance is 25%. The image seems to bring a 25-point gain, but the blind model is 35 points above chance. That is a warning.
Now drop every question the blind model got right (24 of them). That leaves 16 questions. The sighted model still gets 10 of them (7 + 1 + 2), so its accuracy is 62.5%, while the blind model gets 0 of 16. The gain from the image is now 62.5 points. The shrunken benchmark measures vision. Use the device to see where each question sits.
Each square is one question. Filled means answered correctly. Switch between the sighted model, the blind model, and the benchmark after removing every question the blind model got right.
Illustrative: 40 made-up questions with the counts in the text above. Blue: needs the image. Purple: answerable from text alone. Red: leaked into training. Only the arithmetic is exact. Removing every blind-right question also throws out 5 genuine vision questions the blind model guessed; MMStar avoids that cost by adding human review.
Those lucky 5 matter. A blunt filter removes good questions along with bad ones. This is why MMStar’s pipeline has two stages: an automated step pre-selects candidates from existing benchmarks, then people review them. Humans check visual dependency, minimal data leakage, and the need for advanced multimodal capabilities.
The result is MMStar: 1,500 samples chosen by humans, covering 6 core capabilities and 18 detailed axes. The authors test 16 leading vision-language models on it, and apply their two metrics to 7 existing benchmarks. Every sample was reviewed so that the image is needed to answer it.
pythondef audit(model, qs, n_choices=4): sighted = acc(model, qs, show_image=True) # e.g. 0.85 blind = acc(model, qs, show_image=False) # e.g. 0.60 (image blanked) chance = 1 / n_choices # 0.25 leaky = [q for q in qs if answer(model, q, show_image=False) == q.gold] return dict(gain=sighted - blind, excess_blind=blind - chance, suspect=len(leaky) / len(qs), keep=[q for q in qs if q not in leaky])
Symptom. A new model jumps ten points on a benchmark, yet users see no better vision.
Measurement. Run the blind baseline for both models. If the blind score rose too, the jump came from text knowledge or memory.
Fix. Report the sighted score, the blind score and chance together. Remove or rewrite questions that a blind model solves. Our lesson on benchmark auditing covers the general method.
Chapter 7
Find picture pairs that look alike to an encoder but differ to a person, and test perception tasks a child solves in a glance
Look at two photos of a school bus. In the first the bus faces left. In the second it faces right. A five-year-old sees the difference at once. Now ask a top vision-language model, “Which way does the bus face?” You might expect a sure answer. Sometimes the model is confidently wrong, and then explains its wrong answer with invented detail.
The last chapter showed benchmarks that are too easy because text leaks the answer. This chapter shows the opposite kind: tests that are easy for people and hard for models, built on purpose to find where the eyes fail.
Most vision-language models look through an image encoder trained as in Chapter 1: a CLIP-style model. That encoder was trained to match pictures with sentences. A sentence such as “a school bus on a road” is true of both photos. So the encoder has no pressure to keep the two apart, and it may put them almost on top of each other on its map. If two pictures share a spot, everything built on that spot must treat them alike.
The paper Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs turns this into a method. It defines CLIP-blind pairs: images that CLIP perceives as similar despite their clear visual differences. The authors collect such pairs and build MMVP (Multimodal Visual Patterns) from them. The abstract says MMVP exposes areas where state-of-the-art systems, including GPT-4V, struggle with straightforward questions across nine basic visual patterns, often giving incorrect answers and hallucinated explanations.
Here is the logic of a pair. Ask the same question about both images, with opposite correct answers: bus faces left, bus faces right. A system that cannot tell the two images apart must answer the same way for both. So it gets at most one of the two right. The paper also reports a notable correlation: patterns that are hard for CLIP-based models tend to be hard for multimodal models too. That points at the encoder as the source.
Try the logic in a toy. Each row below is a pair of images placed on an embedding axis. If the distance between the two falls inside the blind radius, the encoder cannot separate them. Slide the encoder’s sharpness.
Each row is a pair of pictures with opposite correct answers. The two dots are where an encoder puts them. Inside the shaded zone, the encoder cannot tell them apart.
Illustrative: six made-up pairs with made-up distances and a fixed blind radius. Sharpness multiplies every distance. The paper’s nine visual patterns and its measured accuracies are in the paper; the abstract gives none, so none are quoted here. The ceiling arithmetic (a blind pair can score at most one of two) is exact.
At sharpness 0.60, four of the six pairs fall inside the zone. Each blind pair costs at least one answer out of two, so the best possible score on those twelve questions is 8 / 12 = 66.7%, however clever the language model above the encoder is. This is the idea of a ceiling set by the encoder. The paper’s proposed fix, Mixture of Features, mixes in features from a vision model trained by self-supervision (no captions). The authors report that this can significantly enhance visual grounding.
A second approach converts classic computer-vision tasks into questions. BLINK reformats 14 classic vision tasks into 3,807 multiple-choice questions, paired with single or multiple images and sometimes visual prompts (for example, circled points). Example tasks include relative depth estimation (which of two marked points is closer?), visual correspondence (which point in the second image matches this point in the first?), forensics detection and multi-view reasoning.
json{"task": "relative_depth",
"images": ["scene_0831.jpg"],
"visual_prompt": [{"label": "A", "xy": [0.31, 0.62]}, {"label": "B", "xy": [0.72, 0.48]}],
"question": "Which marked point is closer to the camera?",
"choices": ["A", "B"], "answer": "A"}
The answer key for this record comes from a depth sensor or an existing labeled dataset, not from a person squinting. That is the design trick: borrow ground truth that already exists.
The numbers are stark. Humans average 95.70%. GPT-4V reaches 51.26% and Gemini 45.72%. The abstract says the best models beat random guessing by only 13.17 points (GPT-4V) and 7.63 points (Gemini). Subtract: 51.26 − 13.17 = 38.09 and 45.72 − 7.63 = 38.09. So the random-guess level over the whole set is 38.09%, higher than 25% because many questions have only two or three choices. The gap between a person and GPT-4V is 95.70 − 51.26 = 44.44 points. The authors add that specialist computer-vision models could solve these problems much better.
Symptom. A model reads a page of text in a photo perfectly, then cannot say which of two marked points is nearer.
Measurement. Report accuracy per task, minus the random-guess level for that task, next to human accuracy. One average hides the tasks where a model sits at chance.
Fix. Do not blame the language model first. Probe the encoder with pairs, as MMVP does, and consider adding vision features that do not depend on captions. Our vision-language model lesson shows where the encoder sits in the pipeline.
Chapter 8
Match each test to the question it can answer, then run a six-step audit before you trust a score
A carpenter does not use one tool for every job. A ruler measures length, a level checks a slope, and a square checks a corner. Using the level to measure length gives a number, and the number is nonsense.
Benchmarks are tools in the same way. Each one measures one thing well. The mistake is to quote any benchmark as proof of “vision ability” in general. This chapter sorts the tools, then gives you the checks that decide whether a particular tool is sound.
MMMU asks expert-level questions that need both knowledge and reasoning across many subjects. It is the benchmark for the question, “can this model reason like an expert?” We did not reopen its paper for this lesson. The one fact we use comes from the MMStar abstract, which reports that Gemini Pro scores 42.9% on it without any visual input. That is a reminder that a knowledge-heavy suite can often be attacked from text alone. Our vision-language lesson gives the model background.
MMStar (Chapter 6) is the fair-test tool. It keeps 1,500 human-selected samples whose answers need the image, across 6 core capabilities and 18 axes.
MMVP and BLINK (Chapter 7) are perception probes. MMVP asks straightforward questions across nine basic visual patterns, built from pairs the encoder cannot split. BLINK has 14 reformatted classic vision tasks and 3,807 multiple-choice questions.
CV-Bench comes from the Cambrian-1 paper, which describes a family of multimodal language models designed with a vision-centric approach. Its authors argue that vision design choices are often insufficiently explored and disconnected from visual-representation research. They test more than 20 vision encoders, critically examine existing benchmarks, and introduce CV-Bench as a new vision-centric benchmark. The abstract does not give its size or task list, so we do not. The same paper stresses data source balancing and distribution ratio when curating instruction-tuning data, which is Chapter 3’s idea in a new setting.
Pick a goal below and see which tool fits, and the fact that backs it.
Choose what you want to learn. The matching test lights up on the map. The map places each test by what it mostly measures and by how much text alone can help.
The positions are an editorial judgement for teaching, not a measurement. The numbers in the boxes come from the papers’ abstracts: MMVP (nine visual patterns), BLINK (14 tasks, 3,807 questions), MMStar (1,500 samples, 6 capabilities, 18 axes), MMMU (42.9% for Gemini Pro without images, quoted by MMStar), Cambrian-1 (more than 20 vision encoders).
pythondef run_audit(model, bench, train_index, n_choices=4): out = {"sighted": acc(model, bench), # step 2 "blind": acc(model, bench, blank_image=True), "shuffled": acc(model, bench, shuffle_images=True), # step 3 "chance": 1 / n_choices} dup = [b for b in bench if train_index.max_cosine(b.emb) > 0.95] # step 4 out["contaminated"] = len(dup) / len(bench) out["by_slice"] = {s: acc(model, [b for b in bench if b.slice == s]) for s in set(b.slice for b in bench)} # step 5 return out
Model A scores 70% and model B scores 72% on a 1,500-sample benchmark (the size of MMStar). Is B better? Estimate the noise. For an accuracy p measured on n items, the standard error is √(p(1 − p) / n). For A that is √(0.70 × 0.30 / 1500) = √0.00014 = 0.0118, which is 1.2 points. A rough 95% range is plus or minus 2 standard errors, so A is 70% ± 2.4. The ranges of A and B overlap, so a two-point lead on 1,500 items is weak evidence. If both models also answered the same items, a paired test is tighter, but it still needs the item-level answers. Report the range, not just the point.
Suppose the same two models answered the same 1,500 questions. Many questions are easy for both and hard for both, so those cancel. What matters is the questions on which they disagree. Say A is right and B wrong on 120 questions, and B is right and A wrong on 150. Then B wins by 30 net questions, which is 2 points of 1,500. A simple test asks whether 150 versus 120 looks like a fair coin on 270 disagreements. The coin’s spread is about √270 / 2 = 8.2 questions, so a lead of 15 above the even split is about 1.8 spreads. That is borderline, not strong. The lesson: keep the item-level answers from every run so you can ask this question later.
Symptom. A paper claims state of the art on one benchmark, and a reader cannot reproduce the gain on another.
Measurement. Run all six audit steps on the first benchmark. If blind or shuffled accuracy sits well above chance, the gain was never about vision.
Fix. Pair every headline number with its blind score and its range. Choose a suite for the question you ask, as the device does. Our benchmark auditing lesson goes deeper into the statistics.
Chapter 9
Three quiet failures, each with its symptom, its measurement and its fix: contamination, answer leakage and rare-concept loss
A student sits an exam. Afterward you learn the exam paper was posted online a month before, and the student had read it. Her score is real. What it means has changed. She did not show she can solve the problems. She showed she can remember them.
Models face this situation all the time, because they train on the web and the web holds almost everything, including test sets. This chapter names three failures. Each one fools a careful team, and each one has a check.
Contamination means that test items, or near-copies of them, appear in the training data. The web is the usual route: a benchmark is published on a web page, someone mirrors the images, and the next crawl picks them up.
Symptom. A model scores very high on a benchmark and much lower on a fresh test of the same skill. The MMStar paper reports a related effect: models answer some questions that truly need the image without seeing it, which the authors take to suggest that the samples were memorized during training.
Measurement. Embed every test image. Search the training pool for its nearest neighbour. Plot the highest cosine similarity for each test item. A cluster near 1.0 is a warning. Try it below: 60 toy test images, 12 of them with copies in training. Move the cut and watch two errors trade off.
Each bar counts test images by their highest similarity to any training image. Red: a copy is in training. Blue: clean. Drag the cut: items to its right are flagged.
Illustrative: 60 made-up test images, 12 with training copies. Resized or cropped copies score lower than exact ones, and two clean stock-photo-like images score high, so no cut is perfect. Real systems pair the embedding search with exact hashes and text checks.
Fix. Remove flagged items from the benchmark, or remove the matching images from training before you train. Keep a private held-out set that never goes online. Write the cut and the counts into the report.
pythonD, I = train_index.search(test_emb, 1) # nearest training image per test image flagged = D[:, 0] > 0.90 # cosine cut, tuned by looking at samples print(flagged.sum(), "of", len(test_emb), "flagged") clean_acc = acc[~flagged].mean() # report both numbers flag_acc = acc[flagged].mean()
This is the failure of Chapter 6, seen from the data side. The answer is in the text of the question, the choices or the model’s world knowledge, so no image is needed. The numbers from the MMStar abstract bear repeating: Gemini Pro gets 42.9% on MMMU with no visual input, and Sphinx-X-MoE gets 43.6% with no images, 17.9 points above its language-model backbone.
Symptom. Adding a bigger text model raises the score; improving the vision encoder does not. Measurement. The blind baseline, the shuffled-image test, and the gain between sighted and blind. Fix. Rewrite or drop questions a blind model solves, and review the rest with people.
The last failure is in the training data, not the test. A CLIP-score cut gives every pair the same bar, but the score is a model’s opinion, and it is least sure about rare concepts. So rare pairs fall below the bar more often.
The hero shows it on the toy pool. Keep the top 30% by the CLIP-style score. That is 48 pairs. Of the 140 common-concept pairs, 47 survive, which is 33.6%. Of the 20 rare-concept pairs, 1 survives, which is 5%. The filter is not malicious. It is simply less sure about what it saw less often. The effect is a property of the toy’s set-up, which we built to show the mechanism, and not a measured rate from a real crawl.
Symptom. The final model is good on common objects and poor on rare ones, and nobody can find a bug. Measurement. Count survivors per concept before and after the filter, as in Chapter 3. Compare the survival rate for common and rare groups. Fix. Balance concepts (Chapter 3), use a learned filter trained on clean varied data (Chapter 2), or cut each group separately, which the hero’s “Balanced” button does.
Embedding search finds copies it can recognise. Some contamination hides: a question reworded, an image redrawn, an answer key posted without the image. A second defence ignores how the leak happened. Build a small new test of the same skill after the training cut-off, and compare. If the public score is 25 points above the fresh score, as in the quiz, something helped the public one. Neither check is perfect. Together they give you a fair picture: one finds the copies you can see, the other measures the effect of those you cannot.
Chapter 10
Review the numbers and terms, run the checklist, read the core code, and follow the links onward
You started with eight raw pairs and a hunch that the web is noisy. You now have a method for each half of the problem. For training data: score, filter, balance, remove copies, and compare recipes in a fixed contest. For tests: demand that the image is needed, hide it to check, probe the encoder, and hunt for copies.
| What | Number | Why it matters |
|---|---|---|
| DataComp pool | 12.8 billion Common Crawl pairs; 38 test sets | One shared start line, one shared exam |
| DataComp-1B, ViT-L/14 | 79.2% ImageNet zero-shot; 3.7 points above OpenAI’s ViT-L/14 (75.5% by subtraction) | With model and compute matched, the gap is the data |
| MetaCLIP, ViT-B | 70.8% on 400M curated pairs vs CLIP 68.3%; 72.4% at 1B; ViT-H 80.5% | Balancing over metadata concepts works |
| Data Filtering Network | ViT-H on DFN-5B: 84.4%; DFN-2B has 2 billion examples | A learned filter beats rules; filter quality is not filter accuracy |
| PixMo / Molmo | 3 listed datasets, no external VLMs; 72B model second only to GPT-4o | Strong open models can be built on human-collected data |
| MMStar | 1,500 samples; 6 capabilities; 18 axes; 16 models tested | Image-indispensable, low leakage |
| Blind baselines | Gemini Pro 42.9% on MMMU, no image; Sphinx-X-MoE 43.6%, +17.9 over its backbone | Always run the text-only test |
| MMVP | 9 visual patterns from CLIP-blind pairs | The encoder sets a ceiling |
| BLINK | 14 tasks; 3,807 questions; humans 95.70%; GPT-4V 51.26%; Gemini 45.72%; random 38.09% (by subtraction) | Easy for people, hard for models |
| Cambrian-1 | More than 20 vision encoders; CV-Bench | A vision-centric benchmark and a data-balance lesson |
New terms, in one line each. Image-text pair: a picture plus its words. CLIP score: cosine similarity between the picture and caption embeddings. DFN: a model trained to filter data. Cap: the most pairs one concept may keep. Question coverage: how many checkable questions a caption answers. Blind baseline: accuracy with the image hidden. CLIP-blind pair: two images CLIP sees as similar. Contamination: test items present in training.
pythondef curate(pool, score, keep=0.3, cap=20000): pool = dedup_exact(pool) # Ch 3: hash the image bytes pool = dedup_near(pool, cos=0.95) # Ch 3: embedding neighbours pool.score = score(pool.images, pool.captions) # Ch 1-2: CLIP score or a learned filter pool = pool.top_fraction(keep, per="concept_group") # Ch 9: do not starve the rare return balance(pool, concepts=METADATA, cap=cap) # Ch 3: cap the crowded concepts def trust_score(model, bench, train_index): r = run_audit(model, bench, train_index) # Ch 6, 8, 9 return {"sighted": r["sighted"], "blind": r["blind"], "chance": r["chance"], "contaminated": r["contaminated"], "by_slice": r["by_slice"]}
Every number marked as illustrative in a device is a toy made for this lesson. Every other number comes from the abstract of the paper named beside it.
Now press Present or Teach and explain one half out loud, from memory: why a web scrape needs a filter, how a cap protects rare concepts, why DataComp fixes the model, or how a blind baseline exposes a leaky test. Then go back to the filter at the top and find the setting that keeps the most good pairs and the most rare ones.