VLMs as Bags-of-Words

A vision-language model like CLIP can match a photo to its caption, yet it often cannot tell “the horse is eating the grass” from “the grass is eating the horse”. See why, and a cheap fix.

Learn why training a model to find the right photo for a caption lets it ignore word order, how a 50,000-case test exposes that, and how swapped captions as hard negatives teach order.

Choose a model and a test, then press Ask 20 times. Watch which caption it prefers. Compare CLIP with NegCLIP. Then we build, piece by piece, why one fails and how the other learned to read order.

You need the idea of a vector (a list of numbers) and the idea of a probability (a number from 0 to 1 for how likely something is). We build the rest from zero.

A reading for the MIT MMAI: Multimodal AI course (Spring 2026), tied to the lecture lesson Alignment. Paper by Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky and James Zou (Stanford, ICLR 2023): arXiv 2210.01936.

Pick the true caption

The accuracies are the paper’s: CLIP 59% and NegCLIP 81% on Visual Genome Relation (23,937 cases), CLIP 62% and NegCLIP 71% on Visual Genome Attribution (28,748 cases), where guessing scores 50%. Each ask is a random draw at that accuracy, not a run of the real model. The horse and cat drawings are illustrative.

Chapter 0

A Model That Hears Words but Not Sentences

See why a model can find the right photo for a caption and still miss who is doing what to whom

Picture a photo search box. You type “the horse is eating the grass” and a good photo comes back: a horse, head down, in a field. Now you type “the grass is eating the horse”. A person smiles at the joke. A weak search system returns the same horse in the same field.

That is the whole puzzle of this paper. The two captions use the same words. They mean opposite things. If the system treats them alike, it does not read the sentence. It only checks which words are present.

What a vision-language model is

A vision-language model (VLM) is a model that handles pictures and text together. The best known one is CLIP. It has two parts. An image encoder turns a photo into a list of numbers. A text encoder turns a caption into a list of numbers of the same length.

Such a list of numbers is a vector, and here it is also called an embedding. Think of it as a street address for meaning. Photos and captions that mean the same thing get nearby addresses. To find the best photo for a caption, you look for the address closest to the caption’s address.

This works well. The paper’s opening line says that VLMs score high on dozens of established benchmarks. So the authors ask a sharper question: does that score prove the model understands how things relate in a scene?

The bag of words

A bag of words is a text with its order thrown away. Imagine every word written on a Scrabble tile and dropped into a cloth bag. You can count the tiles. You can say which words are inside. You cannot say which came first.

A reader that sees only the bag treats “the horse is eating the grass” and “the grass is eating the horse” as one and the same text. Both bags hold six tiles: the, the, horse, grass, eating, is. Try it below. Shuffle a caption and watch how many different sentences share one bag.

Shake the bag

The top row is the caption as written. The bottom row is its bag, sorted so order cannot show. Pick a caption, then press Shuffle to see another sentence from the same bag.

The first two captions are from the paper’s Figure 1 and the fourth is its Table 1 example. The count is exact: n words give n! orderings, divided by 2 for each repeated word.

Look at the read-out. A caption of six words has hundreds of orderings. A bag-of-words reader gives all of them one score. Only a few of those orderings are even sensible English, and a smaller few mean something different from the original. The reader cannot tell which is which.

Why the question is fair

You might think no real model could be this blind. The authors show that several famous ones nearly are. They also give a reason, and a fix. The paper makes three claims, and each one gets chapters here.

First, a new test called ARO (Attribution, Relation and Order) shows that state-of-the-art VLMs often sit at or near chance on order and relations. Second, the usual tests did not catch this, because you can do well at photo retrieval with the order scrambled. Third, a small change to training, adding look-alike negatives, teaches the order.

Why care beyond benchmarks? The paper points to text-to-image generators. If a model cannot tell “a black jacket and a blue sky” from “a blue jacket and a black sky”, an image generator built on it may paint the wrong colors on the wrong things. The authors cite earlier work that saw such failures in generated images.

The plain version. A model can match words to pictures without ever learning which word goes with which object. Matching words is cheap. Binding words to roles takes extra pressure from training.

Keep one picture in mind through the whole lesson. A good test separates a reader who knows the roles from a reader who only spots words. If the two get the same score, the test cannot see the difference. Most of this paper is about finding tests that can.

The road

  1. How CLIP learns from pairs: two encoders, a grid of similarities, and a loss.
  2. The ARO benchmark: four tests that swap words and keep the bag.
  3. What the models score: four famous VLMs against chance.
  4. Retrieval without order: scramble the captions and the images, and the score barely moves.
  5. Why training allows the shortcut: the loss does not ask for order.
  6. Composition-aware hard negatives: the fix, with shapes and the loss.
  7. What the fix buys: NegCLIP against CLIP and plain fine-tuning.
  8. Limits and honest checks: what the paper does not show.
  9. Field guide: numbers, checklist, code, and where to go next.

This paper is a reading for the lecture on Alignment, where text and images are pulled into one shared space. This lesson looks at what that space fails to keep.

A model scores “the horse is eating the grass” and “the grass is eating the horse” the same against a photo. What does that show?

Chapter 1

How CLIP Learns From Pairs

Build the grid of similarities and the loss that trains two encoders to agree

Imagine a teacher holding a stack of four photos and four captions, shuffled into two separate piles. The job is to pair them up. The teacher does not say what is in each photo. The only signal is: these two belong together, and the others do not.

That is how contrastive pretraining works. “Contrastive” means the model learns by contrast: it pulls true pairs together and pushes false pairs apart. CLIP trains on hundreds of millions of image and caption pairs scraped from the web, and the paper builds on exactly this recipe.

Two encoders, one space

The paper writes the two parts as functions. The image encoder fi maps an image to a vector in Rd. The text encoder ft maps a caption to a vector in Rd. Both give d numbers. For the CLIP variant used here, ViT-B/32, d is 512 (a fact from the CLIP release, not from the paper).

As data flow, one batch looks like this. N images (N, 3, 224, 224) go through fi and come out as (N, d). N captions as token ids (N, L) go through ft and also come out as (N, d). Each row is then scaled to length 1. After that, the dot product of two rows is their cosine similarity: 1 when they point the same way, 0 when they are unrelated, −1 when opposite.

The grid of scores

Take every image against every caption. That gives a square table. Cell (j, k) holds how well image j matches caption k. The paper calls this matrix S.

Sjk
The score in row j, column k: how well image j fits caption k. The paper’s S is N by N.
fi(Ij) · ft(Tk)
The dot product of the image vector and the caption vector. Multiply matching positions, then add.
‖…‖‖…‖
The length of each vector. Dividing by both lengths leaves only the angle between them.

In code the whole grid is one matrix product: S = img @ txt.T, shape (N, N). The diagonal holds the true pairs. Everything off the diagonal is a false pair, and the batch supplies those for free.

From scores to a loss

Now turn each row into a choice. Row j asks: among all N captions, which one belongs to image j? A softmax turns the row of scores into probabilities that add up to 1: raise e (about 2.718) to each score, then divide each result by the sum of all of them. Bigger scores get bigger shares. The cross-entropy loss then asks one thing: how much probability landed on the correct caption? It is −log of that probability, so it is 0 for a perfect row and grows as the share falls.

Worked example with three pairs. The original CLIP also divides scores by a learned temperature; the paper leaves this out of its text, but we use 0.1 here so the numbers are easy to see. Row 1 holds scores 0.9, 0.2, 0.1. Divided by 0.1 they become 9, 2, 1. Then e9 = 8103, e2 = 7.39, e1 = 2.72. The sum is 8113.1. The share of the correct caption is 8103 / 8113.1 = 0.9988. The loss is −ln 0.9988 = 0.0012.

Row 2 is a worse case: scores 0.5, 0.45, 0.1 where the first is correct. Scaled: 5, 4.5, 1. The exponentials are 148.4, 90.0, 2.72, with sum 241.1. The correct share is 0.6155 and the loss is 0.486. This row still has work to do.

The paper then does the same down each column, for text: given caption k, which of the N images fits? It adds the row losses and column losses. Both directions are cross-entropy.

Train the two encoders to agree

Rows are four images and columns are their four captions. Each cell shows the row probability. Drag Agreement up to make the true pairs (outlined) win.

30%

Illustrative toy: each image and caption is a point on a circle (a 2-number vector). At 100% agreement each caption points at its image; lower values add a fixed random offset. Scores are cosine similarity divided by a temperature of 0.1, as in the worked example above.

Think about what the numbers do to the encoders. When the correct share in a row is low, the loss sends a signal back into both encoders. The image encoder learns to move the photo vector toward its caption. The text encoder learns to move the caption vector toward its photo. Training repeats this across millions of batches until true pairs sit close and false pairs sit far.

Now a failure mode, as symptom, measurement and fix. Symptom: two captions that differ only in word order land at nearly the same spot in the space. Measurement: compare their vectors; a cosine similarity near 1 means the encoder barely tells them apart. Fix: give training a reason to separate them, which is the subject of Chapter 6.

Notice what this task is

Look again at the loss. Its only job is to retrieve: given an image, find its caption among N, and given a caption, find its image. The paper quotes the CLIP authors on this: CLIP pre-trains for the task of image-text retrieval on a noisy web-scale dataset. Hold on to that sentence. Chapter 5 shows why it matters.

One more detail. In CLIP the N captions in a batch are mostly unrelated: a photo of a train, a cooking recipe photo, a beach. Telling them apart is easy. The loss falls fast, without any need to read word order.

In a CLIP batch of N pairs, what does the N by N similarity grid supply to the loss?

Chapter 2

The ARO Benchmark: Swap the Words, Keep the Bag

Build a test where a bag-of-words reader must guess, and see how each test case is made

Suppose you want to find out whether a student reads sentences or only spots keywords. You would not ask a question whose answer is a keyword. You would give two sentences with the same keywords and one meaning flipped. A student who spots keywords can only guess.

That is the design of ARO, short for Attribution, Relation and Order. Every test case gives the model one image and a few captions that share the same words. Only one caption is true. A bag-of-words model scores them all alike and lands at chance.

Why a new test was needed

The paper credits an earlier test, Winoground, which has 400 hand-made cases and covers many skills at once, such as common sense and world knowledge. That makes it hard to say which skill failed, and 400 cases are too few to compare fine-grained sub-types. ARO is built to be large and narrow. It has more than 50,000 test cases, about two orders of magnitude larger than Winoground, and each task isolates one skill.

The four tasks

Visual Genome Relation tests relations such as “sitting on” and “eating”. It has 23,937 cases over 48 relations. A case holds the true caption “the horse is eating the grass” and the swapped caption “the grass is eating the horse”. Chance is 50%.

Visual Genome Attribution tests which property belongs to which object. It has 28,748 cases over 117 attribute pairs, such as gray and wood. The true caption “the crouched cat and the open door” faces the swap “the open cat and the crouched door”. Chance is 50%.

COCO Order and Flickr30k Order test order in general. The model sees one image, the real caption, and four scrambled versions of it. That is five options, so chance is 20%. Table 1 of the paper lists the four scrambles: shuffle the nouns and adjectives; shuffle everything but the nouns and adjectives; shuffle three-word chunks (trigrams); shuffle the words inside each trigram.

How a Relation case is built

The data comes from Visual Genome, a set of over 100,000 images where people marked objects, their attributes and their relations. The authors use its scene graphs through the GQA annotations. A scene graph is a list such as: object 1 is a horse, object 2 is grass, relation is “eating”. The recipe in Appendix A has six steps.

  1. Walk the scene graphs. Keep objects at least one quarter of the image wide and one quarter tall, so they are easy to see.
  2. For Relation, pair two objects of different categories (not two dogs). For Attribution, pair two objects that each have an attribute, with both objects and both attributes different.
  3. Cut the smallest box that contains both objects, to cut distraction from a busy scene.
  4. Fill the template. Relation true: “the [object 1] is [relation] [object 2]”. Swapped: “the [object 2] is [relation] [object 1]”.
  5. For Attribution, true: “the [attribute 1] [object 1] and the [attribute 2] [object 2]”. Swapped: the two attributes trade places.
  6. Drop symmetric relations such as “near” or “next to”, because swapping them changes nothing.

Step 6 matters. If the relation is symmetric, both captions are true and the test is unfair. The same care explains why the Order tasks use spaCy, a tagging tool, to find nouns and adjectives before shuffling.

How a model is scored

Scoring is plain retrieval turned around. For one image, compute the similarity of the image with each caption option. The model’s answer is the option with the highest similarity. Accuracy is the share of cases where that option is the true one.

The paper reports macro accuracy for the Visual Genome tasks: accuracy is computed per relation (or per attribute pair), then averaged. That stops a common relation like “on” from drowning out a rare one.

Here is a small worked count, with illustrative numbers. Say a scene graph lists three large objects of different categories: a horse, a fence and the grass. That gives three pairs. If the graph also records a relation for each pair, such as “horse eating grass”, each becomes one test case with a true and a swapped caption. Run this over every image and you reach 23,937 cases for Relation.

The recipe also protects against a quiet failure. Symptom: a model scores well on a set of swapped captions. Measurement: check whether the swapped caption could be true too. Fix: remove such pairs before scoring. That is the job of step 6, and of the size filter in step 1, which stops tiny, unclear objects from turning a test of order into a test of eyesight.

Take the test

Choose a task, read the photo description, and pick the true caption. Then see what a bag-of-words reader sees.

The captions are the paper’s examples (Figure 1 and Table 1). The photo descriptions are ours, written to match each caption.

Notice what you did. You knew the answer in a second. A person finds this trivial. A model that scores all options alike has no way to pick better than luck, and the paper’s next question is how far real models sit above luck.

Why does the Order task have a chance level of 20% while the Relation task has 50%?

Chapter 3

What Four Famous Models Score

Read the ARO results against chance and learn what a score below chance means

Four students sit the same exam. The exam has two-choice questions. A coin flip scores 50%. Before you read anyone’s mark, you should draw a line at 50%. A mark of 59% is then a story: barely better than a coin.

The authors test four state-of-the-art VLMs. CLIP is the ViT-B/32 model. BLIP and X-VLM are the Base versions. For the Visual Genome tests the authors use the variants fine-tuned on COCO. Flava is the “flava-full” model from the Hugging Face library. All checkpoints come from the released repositories.

Accuracy, written out

The score for one task is a plain average over M test cases. For case m with image Im and options Tm1 to TmK, where option 1 is the true one:

Acc
The share of cases where the model picks the true caption. Chance is 1/K: 50% with two options, 20% with five.
1/M
Average over all M test cases. For Relation, M is 23,937.
𝟙[…]
One when the highest-scoring option is option 1, the true caption, and zero otherwise. S is the model’s image-text similarity.

What the bars say

Turn to the device. On Relation, CLIP and BLIP both land at 59%, and Flava at 24%. X-VLM reaches 73%. Only one model is clearly useful, and it still misses more than a quarter of simple cases.

Attribution looks better: BLIP 88%, X-VLM 87%, Flava 73%. CLIP stays at 62%, close to chance. So models that look weak on one test can look strong on another. The Flava model shows it best: 73% on Attribution but 24% on Relation.

Read the scores against chance

Choose a task. The dashed line is chance. Bars that stop short of the line are worse than guessing.

Numbers are the paper’s: Visual Genome results from Table 2 and the text of Section 2.3, Order results from Table 5 (mean of 3 seeds).

A short worked example shows why macro accuracy matters. Suppose relation A has 100 cases and a model gets 90 right. Relation B has 10 cases and the model gets 5 right. The plain average over all 110 cases is 95 / 110 = 86%. The macro average is (90% + 50%) / 2 = 70%. The macro number does not hide the weak relation behind the big one. Table 2 of the paper lists all 48 relations one by one for this reason.

Now the failure mode, as symptom, measurement and fix. Symptom: CLIP is near chance on spatial words such as “behind” and “below”. Measurement: Table 2 gives CLIP 56% on spatial relations, with single relations such as “to the left of” at 49%. Fix: Chapter 6 trains with swapped captions that make exactly this kind of mistake costly.

The data flow of one test case is short. One image goes through the image encoder and gives a vector of shape (d,). Each option caption goes through the text encoder and gives its own (d,) vector. The dot products give K scores, one per option. The largest score wins. There is no training in this step: the models are used as released.

The order tasks

On Order the picture is worse. Chance is 20%. On Flickr30k Order, CLIP gets 59.5%, X-VLM 47.3%, BLIP 36.9%, and Flava 12.9%. On COCO Order the numbers fall further: CLIP 46.0%, X-VLM 36.2%, BLIP 32.1%, and Flava 3.9%.

Below chance is information

Flava’s 3.9% on COCO Order is not noise. A coin flip would score 20%. To score 3.9%, a model must reliably prefer a scrambled caption over the real one. It picks the original in about 1 case in 26.

That tells you the model has a systematic bias. Perhaps scrambled captions put the nouns and adjectives in places that match some habit of its text encoder. The paper does not explain the cause. The measurement still carries a clear signal: this model does not treat the original as special.

Spatial words and verbs

The authors also break the Relation task down. BLIP scores 66% on spatial relations like “behind” and 56% on verbs like “eating”. CLIP is the reverse: 56% on spatial and 61% on verbs. So CLIP and BLIP each sit near chance on one family and only a little above it on the other. X-VLM does better on both, at 74% and 73%.

The paper’s Figure 1 gives one vivid case: BLIP, shown the horse photo, picks “the grass is eating the horse” with 81% probability.

One more reading aid: the error bars in the paper’s figures come from 5 seeds for the Order tasks, since the scrambles are random. (The paper’s text says 5 seeds, and its appendix Table 5 says 3 seeds for the same task; we report the table’s means.) The Visual Genome tasks use fixed swaps, so they have no such randomness. When two bars sit within a point or two, treat them as tied.

A hint about generators

The authors link this to image generators. Some text-to-image models use CLIP as the text encoder and struggle with relations. Imagen uses T5, a language model, as its text encoder and does better on compositional tests. The authors only speculate that the language model prior helps. It is a hypothesis, not a result of this paper.

Flava scores 3.9% on COCO Order, where chance is 20%. What is the best reading?

Chapter 4

Retrieval Works Without Word Order

Scramble captions and images, and see why the usual tests never caught the problem

A librarian is asked to find the book with a given blurb. She sees the blurb with all its words shuffled: “books looking at people are the”. She still finds the right book, because no other blurb in the library uses those words. The shuffled blurb is enough.

That is the second claim of the paper. If standard tests can be passed with scrambled captions, then a good score on them cannot show that a model reads order. The tests were blind to the problem.

The standard test: retrieval

The usual VLM test is image-text retrieval. Take a pool of images and captions. For one query, rank the other side by similarity. The score is Recall@K: the share of queries whose true match appears among the top K results. Recall@1 asks for the very first result. Recall@5 allows five.

The paper uses two datasets, COCO (5,000 test images) and Flickr30k (1,000 test images), with the Karpathy splits that earlier work used. Captions in both are full of structure. If any dataset needed order, these would. For these runs the authors use the BLIP and X-VLM checkpoints fine-tuned on the matching dataset, and CLIP as released.

The experiment: remove the order, keep the rest

The authors change only the test data, never the models. They make two kinds of change.

Scramble the captions with the Table 1 strategies. “The grass is eating the horse” becomes “eating the grass horse the” when all words are shuffled. Other strategies move only nouns and adjectives, or move three-word chunks, or shuffle inside each chunk.

Scramble the images. Editing objects is hard, so they use a blunt tool: cut the image into 4 rows, 4 columns or 9 patches, and shuffle the pieces. An object that was below another can now be anywhere. Each experiment runs with 3 random seeds.

Then the same pretrained models are scored again with the scrambled data. If the models needed order, their scores would collapse.

What happened

They did not collapse. Take CLIP on COCO, finding the caption: Recall@1 was 50.3% with intact captions. With every word shuffled it was 34.1%. That is a drop of 16.2 points, and CLIP keeps 68% of its score. BLIP goes from 81.4% to 69.0%.

Image scrambling is a little harsher but tells the same story. CLIP’s COCO Recall@1 goes from 50.3% to 40.2% with shuffled rows and 28.4% with 9 shuffled patches. Even then, it finds the right caption far more often than chance, which on 5,000 images would be well under 1%.

A tiny worked example makes Recall@K concrete. Say a pool holds four captions and one image. A model ranks them: caption C first, caption A second, the true caption B third, D fourth. Recall@1 is 0 for this query, since B is not first. Recall@5 would be 1, since B is in the top five. Over thousands of queries, Recall@1 is the share of queries where the true match came first.

The image-side trick has a precedent. The paper cites Qin and colleagues, who used patch shuffling to show that vision transformers are insensitive to patch-based changes in classification. The authors reuse the idea for retrieval, because it removes the layout cues that relations depend on.

Scramble it and re-score

Pick a dataset and what to scramble. Drag the slider to choose how. The pale bar is the intact score; the solid bar is after scrambling.

Numbers are the paper’s Tables 3 and 4: the “Text Recall@1” column, mean of 3 seeds.

It helps to see what each scramble removes. Shuffling nouns and adjectives keeps the small words in place but breaks which adjective belongs to which noun. Shuffling everything but nouns and adjectives keeps the content words in place and breaks the glue between them. Shuffling trigrams keeps local phrases intact and breaks the long-range order. Shuffling inside trigrams does the reverse. Because all four leave the set of words unchanged, a bag of words cannot see any of them.

Reading this honestly

The scores do fall. A drop of 10 to 16 points is not nothing. But a model that really needed order would fall toward chance, and these models stay far from it. The authors call the loss “marginal”. A careful reader would say it is moderate, and that is the point: moderate loss means most of the retrieval skill is available without order.

Why? In a pool of thousands of images, the words “books”, “people” and “looking” already narrow the choice. Few other images have the same nouns together. Order adds a little help at the top of the ranking, and not much more.

Worked check with the table. CLIP, Flickr30k captions: 78.4% intact, 58.7% with all words shuffled, 68.0% with words shuffled inside trigrams. That keeps 75% and 87% of the intact score. A model blind to words would be near 0.1%, since 1,000 images give a chance of 0.1% at Recall@1.

What this changes. A benchmark can be passed by a shortcut. If a test accepts a bag of words, a high score on it says nothing about word order. ARO exists for this reason.
CLIP keeps about 68% of its COCO Recall@1 with all caption words shuffled. What is the right conclusion?

Chapter 5

Why Training Never Asks for Order

See how a loss that is already near zero teaches nothing, and why batches of unrelated captions make that happen

A horse named Clever Hans seemed to do arithmetic. He tapped out answers with his hoof. In truth he watched his trainer’s face and stopped when the trainer relaxed. He found a shortcut that earned the reward without the skill. The paper uses the same idea, called shortcut learning: a network finds a cheap rule that gets the reward and never learns the harder skill.

The authors argue that bag-of-words is such a shortcut for contrastive training. It is a hypothesis, and they say so. The evidence is Chapter 4: the retrieval task can be solved without order, so the loss can be made low without order.

Where the learning signal comes from

Recall the row loss from Chapter 1: −log of the probability the model gives the true caption. Learning happens through the gradient of this loss. For the true caption’s score strue, the gradient has a very simple form.

∂ℓ / ∂strue
How much the loss changes when the true score moves up a little. A big negative number means a strong push to raise the score.
ptrue
The softmax share the model gives the true caption. When it is near 1, the push is near 0.

This is the heart of the chapter. If the model already puts 99% on the true caption, the push is 1%, and training moves on. The model keeps learning only from rows where the share is low. Rows where it already wins give almost nothing.

A toy that makes the shortcut visible

Take one image with three content words in view: horse, grass, eating. Build a bag-of-words scorer. It gives 2 points for every content word of the caption that also appears in the image. The true caption “the horse is eating the grass” has three matches, so its score is 6.

First batch: the true caption plus three unrelated captions, such as “a cat on a desk”. Each scores 0. The shares come from e6 = 403.4 against three copies of e0 = 1. So ptrue = 403.4 / 406.4 = 0.9926. The loss is −ln 0.9926 = 0.0074. The push on the true score is 0.0074. The model wins without trying.

Second batch: swap in the hard case. The true caption, the swapped caption “the grass is eating the horse”, and two unrelated captions. The swap has the same three matches, so it also scores 6. Now ptrue = 403.4 / (403.4 + 403.4 + 2) = 0.4988. The loss is 0.695, close to ln 2, the loss of a coin flip. The push is 0.50, about 68 times stronger than before.

Now give the model some order sense. Add w extra points to the true caption when the roles are right. With w = 4 the true score is 10, and in the hard batch ptrue = 22026 / (22026 + 403 + 2) = 0.982, loss 0.018. That is far below 0.695, and only an order-aware scorer reaches it.

Feed it easy and hard batches

Choose the foils in the batch. Then raise Order bonus to give the scorer a sense of word order. Watch the loss and the push.

0

Illustrative toy of ours, not a result from the paper. The bag scorer gives 2 points per matching content word and the order bonus is added only to the true caption.

What this says about real data

Large pretraining sets are built to cover a huge range of concepts. A batch of 32,000 random web pairs has very few captions that use the same words as each other. The paper puts it this way: the datasets are not designed to contain many images with similar-word captions that must be told apart. So the “swapped foil” case almost never appears.

In that setting, behaving like a bag of words becomes, in the authors’ words, a high-reward strategy. The model gets a low loss, so nothing makes it change.

The authors also note why earlier hard-negative tricks did not help. BLIP and an earlier model, ALBEF, mine negatives by picking pairs with high similarity. The paper says this alone does not seem enough to teach relations and word order. Similar-looking items need not differ in order.

The same pattern shows up outside vision. The paper’s related work lists findings from language models. Pretraining BERT on sentences with shuffled words only marginally hurts downstream tasks. Some GLUE tasks can be solved while ignoring word order. On the vision side, a bag-of-local-features model performs almost as well as its state-of-the-art rivals on ImageNet. And another group trained contrastive models with only a bag of words in place of the caption and saw no major drop in zero-shot classification. Each result says the same thing: if a benchmark allows a shortcut, models take it.

As data flow, this means the text encoder can produce a useful vector of shape (d,) from the presence of a few words alone, such as horse, grass and eating. Nothing in the loss asks the encoder to also write the roles into that vector. So it often does not.

The core claim. Contrastive training optimises retrieval. Retrieval can be solved without order. So nothing in the loss rewards order, unless the batch is built to demand it.
In the toy, why does the swapped foil make training far more informative than unrelated foils for a bag-of-words scorer?

Chapter 6

The Fix: Composition-Aware Hard Negatives

Add swapped captions and look-alike images to each batch so the loss finally asks for order

A driving examiner who only tests on empty straight roads will pass drivers who cannot turn. To teach turning, the examiner must add turns to the test. The paper does the same for CLIP. It cannot change what the web contains, but it can change the batch.

A hard negative is a wrong answer that looks a lot like the right one. The paper adds two kinds, and the word composition-aware means the wrong answers are built by changing how words and objects are composed, not by picking random mismatches.

Ingredient 1: negative captions

For each image-caption pair, make a new caption that uses the same words with two parts swapped. The part can be a noun phrase, a noun, an adjective, an adverb or a verb phrase. Take “The horse is eating the grass and the zebra is drinking the water”. Swapping nouns gives “The zebra is eating the grass and the horse is drinking the water”. Swapping verb phrases gives “The horse is drinking the grass and the zebra is eating the water”.

The paper builds these from the COCO captions with spaCy, a language tool that tags each word. For each caption it builds up to 5 possible negatives. Noun phrases only count if they have three tokens or more, so they do not overlap with noun swaps. A caption with no possible swap is dropped. At each epoch, one negative is sampled for each caption.

Ingredient 2: strong alternative images

The second ingredient works on the image side. Use CLIP to compute the similarity between all pairs of COCO images. For each image, keep its 3 nearest neighbours. At each epoch, sample one of the three and add that image, with its caption and its negative caption, to the batch. Now the batch holds two photos of nearly the same scene, and the model must tell which caption goes with which.

The new loss

Recall the grid S of Chapter 1. With negatives, the caption side gets twice as wide. Let T be the N true captions and T− the N negative captions. Stack them into 2N captions. The new grid has N rows and 2N columns.

img (N, d) @ [txt; neg].T (d, 2N) = S̃ (N, 2N)

Lrow
Image to caption. Each image must pick its true caption among all 2N captions, including the N negatives. Column j of the true block holds the right answer.
Lcol
Caption to image, for the N true captions only. A negative caption has no matching image, so it adds no column term, as the paper states.

The paper describes this in words, with the cross-entropy computed “as in CLIP”. The symbols above are the standard CLIP form with the change it describes. Actual CLIP also scales the scores by a temperature.

Why it works: a worked row

Take one image whose true caption scores 8 after scaling. Its negative caption (the swapped one) scores 7, because it has the same words. Two other captions score 1 and 2. Without the negative: e8 = 2981, e1 = 2.7, e2 = 7.4. The true share is 2981 / 2991 = 0.9966 and the loss is 0.0034. With the negative: e7 = 1097 joins the sum, which becomes 4088. The share drops to 0.729 and the loss rises to 0.316. The model must now push the true score above the swap, which can only happen if it separates the two by word order.

Grow the batch

Pick a batch size N, then turn the two ingredients on and off. The grid shows rows (images) against columns (captions). Pink columns are negatives.

1,024

The paper fine-tunes with N = 1,024. Neighbour images add one extra photo per image, so rows and columns double: this is our reading of the paper’s description, and the grid counts are our arithmetic.

Look at what the swap does to the data flow. The text encoder receives the negative caption exactly as it receives a true one: token ids of shape (N, L) become a vector of shape (d,). Nothing about the encoder changes. Only the loss changes, because the negative vector now sits among the candidates. That is why the fix is cheap: no new layers, no new parameters, one extra text pass per caption, and more image passes when neighbours join.

Why mine negatives from the dataset instead of drawing random captions? Random captions are easy, as Chapter 5 showed. Swaps keep every word and change only roles, so the only way to win is to use the roles. Neighbour images do the same on the other side: two photos of nearly the same scene differ in fine detail, so coarse word matching no longer works.

The training recipe

Compute was the limit, so the authors fine-tune a pretrained CLIP ViT-B/32 on the COCO training split rather than train from scratch. They run 5 epochs, try learning rates 1e-5, 5e-6 and 1e-6, and pick by retrieval on the COCO validation split. They use AdamW, 50 warmup steps, a cosine learning-rate schedule and N = 1,024 on one NVIDIA RTX 2080 Ti. The model they get is named NegCLIP. A control, CLIP-FT, is fine-tuned the same way without the hard negatives.

Why does the column-wise loss skip the negative captions?

Chapter 7

What the Fix Buys, and What It Costs

Compare CLIP, plain fine-tuning and NegCLIP on four order tests and seven everyday tasks

You change a recipe and the cake rises. Before you credit the new ingredient, you must check two things. Would the cake have risen anyway, with the oven change alone? And did the new ingredient spoil the taste? The paper runs both checks.

Three models, two kinds of task

The three models are CLIP (original), CLIP-FT (CLIP fine-tuned on COCO with no hard negatives) and NegCLIP (CLIP fine-tuned on COCO with hard negatives). CLIP-FT is the control. It separates the effect of extra COCO training from the effect of the negatives.

There are two sets of tasks. The compositional set is the four ARO tasks. The downstream set checks that NegCLIP is still a good CLIP: CIFAR10 and CIFAR100 and ImageNet for classification, plus Flickr30k and COCO retrieval in both directions.

The compositional gain

NegCLIP raises VG-Relation from 59% to 81%, VG-Attribution from 62% to 71%, COCO Order from 46% to 86%, and Flickr30k Order from 59% to 91%. The Order tasks gain 40 and 32 points. Those are the tasks the training negatives most resemble, since both involve swapped words.

In the paper’s text, the Relation starting point is quoted as 63%. Tables 2 and 6 list CLIP at 59% and CLIP-FT at 63%. We use the tables. Either way NegCLIP’s 81% is the best Relation score of all models tested, above X-VLM’s 73%.

On Attribution, the paper calls NegCLIP comparable to X-VLM and BLIP. The numbers are 71% against 87% and 88%, so a gap remains. The method closes much of the distance from CLIP’s 62%, not all of it.

The control that matters

Now look at CLIP-FT. After the same fine-tuning without negatives, it scores 63% on Relation and 65% on Attribution, a small gain. On the two Order tasks it falls: Flickr30k Order from 59% to 50%, COCO Order from 46% to 36%. So more COCO training alone does not teach order. It slightly hurts it. The hard negatives did the work.

Compare the three models

Choose a group of tasks. Compare the three bars on each task. Toggle the control off to see just CLIP against NegCLIP.

Numbers are Table 6 of the paper, rounded to whole percent as printed. Accuracy for classification and Order tasks, Recall@1 for retrieval, macro accuracy for Visual Genome. The column order in the extracted table was read as CLIP, CLIP-FT, NegCLIP and checked against the paper’s text.

The arithmetic of the Order result is simple. On Flickr30k Order, five options give chance of 20%. CLIP gets 59%, which is 2.95 times chance. NegCLIP gets 91%, which is 4.55 times chance and only 9 points from perfect. On COCO Order the same ratios are 2.3 and 4.3. The weakest result before the fix is now the strongest of all the models in the paper’s tests on these two tasks.

The cost side

On the downstream tasks, NegCLIP stays close to CLIP. CIFAR10 goes from 95% to 94%, CIFAR100 from 80% to 79%, ImageNet from 75% to 72%. A 3-point loss on ImageNet is the biggest cost. On retrieval, NegCLIP gains: COCO text Recall@1 rises from 50% to 56%, and Flickr30k image Recall@1 from 59% to 67%. Much of that gain comes from the COCO fine-tuning itself, since CLIP-FT gains as much or more.

Where the gain lands

Table 2 splits Relation by type. NegCLIP gets 66% on spatial relations (CLIP 56%) and 86% on verbs (CLIP 61%). The verbs gain the most. The training negatives swap verb phrases and nouns. Our reading, not the paper’s claim: the gain probably depends on how closely the negatives resemble the test.

The authors are modest about the claim. They say NegCLIP is not the best possible model for relations. Their point is that targeted, cheap negatives bring large gains on composition without hurting everyday tasks. They also raise a question: whether simply scaling pretraining data would be as efficient as such algorithm changes is unclear.

One detail about how the models were chosen matters for trust. The authors picked the learning rate by retrieval performance on the COCO validation split, not by the ARO scores. So the compositional tests were not used to tune NegCLIP. That makes the gains a fair read of the method, as far as one run can be.

Here is a worked reading of the cost and the gain together. On the four compositional tasks NegCLIP gains 22, 9, 32 and 40 points over CLIP, an average of 25.75 points. On the three classification tasks it loses 1, 1 and 3 points, an average of 1.67 points. The trade is lopsided in favour of the fix, but it is not free. Where a product depends on ImageNet accuracy, the 3 points deserve a second look.

The failure mode to watch for is the gap on Attribution. Symptom: 71% against X-VLM’s 87%. Measurement: Table 6 against the Figure 1 numbers. Fix: unknown from this paper. Swapping adjectives and nouns helped, but other ingredients in BLIP and X-VLM may matter, and the paper does not test them.

What we can say. On these tests, hard negatives gave large compositional gains, plain COCO fine-tuning did not, and everyday tasks lost at most 3 points.
Why does the paper include CLIP-FT?

Chapter 8

Limits, Blind Spots and Honest Checks

Separate what the paper shows from what it only suggests, and learn the checks to run on any similar benchmark

A good bridge inspector does not only say the bridge stands. She says what load was tested, which bolts were not reached, and what she would test next. This chapter does that for the paper. It has two parts: the limits the authors state, and checks we suggest ourselves. We mark the second part clearly.

Limits the paper states

Fine-tuning, not pretraining. The authors could not train a model from scratch. NegCLIP is CLIP fine-tuned on COCO. They expect the benefits to carry to contrastive pretraining and say future work must check.

Small batch. The original CLIP used N = 32,000. The authors used N = 1,024 on one GPU. They expect larger batches to give further gains. That is an expectation, not a measurement.

One run for the fix. The CLIP and NegCLIP experiments were not repeated across seeds, because of compute. The training seed is fixed, so others can replicate it. The benchmark scores of the four existing models use 3 seeds where the perturbation is random.

Only contrastive learning. The paper studies contrastive pretraining. How other objectives interact with composition is listed as future work.

Datasets inherit old flaws. ARO builds on Visual Genome, COCO and Flickr30k. The ethics statement notes these data have known privacy and stereotype problems and that the paper continues their use for comparison. It also notes that when models fail at composition they may fall back on stereotypes. It cites the example of an image generator that gave a property to the wrong person in a prompt about a meeting leader. Better models may also help surveillance uses, which the authors call out as a responsibility.

Checks we suggest (our reading, not the paper’s)

Is the swapped caption plausible? Look at ARO’s swaps: “the grass is eating the horse”, “the open cat and the crouched door”. Such captions are odd English. A model that only knows which sentence sounds normal could win without looking at the image at all. A good benchmark should include a text-only baseline: score the options with no image, and report it beside the model.

Try it. In the device below, pick the likelier caption using words alone. Our own rating of how obvious each is appears afterwards.

Guess with no photo

For each pair, click the caption that sounds more likely as everyday English. You cannot see any photo.

The pairs are the paper’s examples (Figures 1 and 3, Section 4 and Appendix A). The “obvious from text” labels are our own judgement, not a measurement from the paper.

If most pairs are obvious from the words, a high ARO score could come partly from a language prior, which means the model leans on which sentence sounds normal instead of on the photo. Relation cases with a symmetric pair or two plausible readings, like the dog and the tree, are the informative ones. They need the photo.

Does the test look like the training negatives? NegCLIP is trained on swapped nouns, adjectives and verb phrases, and ARO tests swapped nouns and attributes. Gains on a test that resembles the training trick are real, but they may not carry over to a different kind of composition, such as counting or negation.

What would the checks look like? A text-only baseline score. A test set built with foils that are equally plausible. A breakdown by relation type, as the paper itself does in Table 2. And a second seed for any small claim.

Is the benchmark about composition alone? The authors say that the earlier Winoground set mixes many skills, so a failure there is hard to pin on composition. ARO fixes that by isolating relations, attributes and order. A related paper, cited in the text, argues that Winoground has further challenges beyond composition. The lesson for any reader: when you build or pick a test, decide which single skill it should isolate, and then check that it does.

How should you report one number? The paper averages the four existing models over 3 seeds where the scramble is random, and reports error bars. For small differences, such as two points on Attribution, a single run is not enough evidence. For large ones, such as 46% to 86%, it is.

Does the result depend on the swap method? The negatives come from spaCy tags. If the tagger mislabels a word, the swap may produce a caption that is still true, such as swapping two synonyms. The paper does not report how often this happens. A reader who reuses the code should sample a few hundred negatives and read them by eye.

Failure, measurement, fix

Symptom: a model scores 90% on a compositional test. Measurement: score the same options with the image removed. If the text-only score is also high, the test is leaking. Fix: rebuild the foils so that both captions are equally likely English, then re-run the model.

Which extra measurement would best test whether an ARO score really depends on the image?

Chapter 9

Field Guide: The Paper on One Page

Keep the numbers, the terms, the checklist and the core code, and find where to read next

The paper makes one argument in three steps. Vision-language models often behave like bags of words on tests of relations, attributes and order. Standard retrieval tests hide this, because retrieval can be solved without order, and contrastive training optimises retrieval. Adding swapped captions and look-alike images to each batch teaches order cheaply, with little cost on everyday tasks.

The numbers

WhatValueSource
ARO sizeover 50,000 cases: 23,937 relation, 28,748 attributionSection 2.1
Chance levels50% (Relation, Attribution), 20% (Order)Sections 2.1 and 2.3
CLIP Relation, Attribution59% and 62%Tables 2 and 6
Lowest scoreFlava, COCO Order, 3.9% (chance 20%)Table 5
CLIP COCO Recall@150.3% intact, 34.1% all words shuffledTable 3
CLIP COCO Recall@1, 9 patches28.4%Table 4
NegCLIP vs CLIPRelation 59 to 81, Attribution 62 to 71, COCO Order 46 to 86, Flickr30k Order 59 to 91Table 6
CLIP-FT controlCOCO Order 36%, Flickr30k Order 50%Table 6
Fine-tuningCOCO, 5 epochs, AdamW, 50 warmup steps, cosine, N = 1,024, one RTX 2080 TiAppendix C.3
Negatives per captionup to 5 swaps, one sampled per epoch; 3 nearest neighbour images, one sampled per epochAppendix C.1, C.2

The terms

TermPlain meaning
Bag of wordsA text with its word order removed.
Contrastive lossCross-entropy that pulls true image-caption pairs together and pushes the rest apart.
Recall@KShare of queries whose true match is in the top K results.
Hard negativeA wrong answer that looks a lot like the right one.
Macro accuracyAccuracy per relation or attribute pair, then averaged.
Shortcut learningA cheap rule that earns the reward without the intended skill.

The checklist

Tick these as you finish. They are the questions to ask of any new VLM paper.

The core as code

The loss and the negative builder, in PyTorch style. The paper gives the objective in words; this is the standard CLIP form with its change. The swap routine simplifies the paper’s spaCy recipe to nouns, adjectives and adverbs.

pythonimport torch, torch.nn.functional as F

def negclip_loss(img_f, txt_f, neg_f, logit_scale):
    # img_f, txt_f, neg_f: (N, d), each row L2-normalised. logit_scale = 1 / temperature.
    N = img_f.shape[0]
    all_txt = torch.cat([txt_f, neg_f], dim=0)              # (2N, d): true block, then negatives
    S = logit_scale * img_f @ all_txt.T                        # (N, 2N)
    labels = torch.arange(N)                                   # true caption of image j is column j
    loss_row = F.cross_entropy(S, labels)                      # each image picks among 2N captions
    loss_col = F.cross_entropy(S[:, :N].T, labels)             # true captions only: no column term for negatives
    return (loss_row + loss_col) / 2

def swap_negative(caption, nlp, rng):
    doc = nlp(caption)
    kinds = [[t.i for t in doc if t.pos_ == p] for p in ("NOUN", "ADJ", "ADV")]
    kinds = [k for k in kinds if len(k) >= 2]
    if not kinds: return None                          # no swap possible: drop this caption
    i, j = rng.sample(rng.choice(kinds), 2)
    toks = [t.text for t in doc]
    toks[i], toks[j] = toks[j], toks[i]
    return " ".join(toks)                                   # "the grass is eating the horse"

# neighbours: top-3 most similar COCO images by CLIP similarity, one sampled per epoch
# sims = img_all @ img_all.T ; sims.fill_diagonal_(-1) ; nbrs = sims.topk(3, dim=1).indices

Where it sits, and where to go next

The takeaway. A score is only as informative as the test is hard to cheat. Retrieval did not demand order, so CLIP did not learn it. Once the batch demanded it, CLIP learned it.

Now press Present or Teach and explain, from memory, why a model can ace retrieval and still fail “the grass is eating the horse”. Then go back to the top and ask the two models 200 times each.

Credit: this lesson teaches the paper by Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky and James Zou (Stanford University), published at ICLR 2023, arXiv 2210.01936. All figures of results come from that paper. Toy devices and the checks in Chapter 8 are ours.

When and Why Vision-Language Models Behave like Bags-of-Words, and What to Do About It?
Read the paper (arXiv 2210.01936) · The authors’ code · MIT MMAI schedule · The Alignment lecture lesson · Back to Gleams