A picture and its caption can agree, clash or say nothing to each other. Three simple questions sort any pair into one of eight classes, and a network learns the sorting at 80.8% accuracy.
Learn how Otto, Springstein, Anand and Ewerth turn decades of picture-and-word theory into a classifier a search engine can use, and how they built the training data without a crowd of annotators.
Set the three dials: does the pair share things, do they agree, and which side carries the message? Watch the eight classes light up, and the ten settings that cannot exist stay dark. Then we build the whole system, piece by piece: the theory, the data, the network and the results.
You need a feel for what a caption is and a rough idea of a probability. We build the rest from zero, including what an embedding is.
A reading for MIT MMAI, Multimodal AI (Spring 2026, Paul Liang), paired with Lecture 7, Alignment. Paper: C. Otto, M. Springstein, A. Anand, R. Ewerth, “Understanding, Categorizing and Predicting Semantic Image-Text Relations” (arXiv 1906.08595). The course lists the journal version, “Characterization and classification of semantic image-text relations” (IJMIR 2020). This lesson follows the arXiv text, the same authors’ earlier version. The examples and toy numbers are ours and are labelled where they appear.
The grid is the paper’s Figure 4: 2 overlap levels × 3 agreement levels × 3 status levels = 18 settings, of which 8 are classes. The two example boxes are our own toy pairs written in the spirit of the paper’s examples, not images from the paper.
Chapter 0
See why a search engine must know how a picture and its words relate, and not only whether they match
You open a news page. Under a headline about city birds sits a photo of a grey pigeon. The text beside it says: “This bird is known for its bright, colourful plumage.” Something is wrong, and you feel it in under a second. You do not need to read the article. The picture and the words disagree.
Now change one thing. The photo shows a red parrot, and the text is the same. The page feels fine. Change it again: the photo shows a blue and yellow parrot. Still fine. The words did not change at all, yet the same sentence went from wrong to right when only the picture moved.
You just did a job that a computer finds very hard. You judged how a picture and a text relate. Not “what is in the photo?” and not “what does the sentence say?”, but “do these two belong together, and in what way?”
Most web pages mix pictures and words. A modality is one channel of information: text is one modality, an image is another, audio is a third. When a search engine answers a query, it usually shows the text results in one list and the images in another. It does not know whether a given photo explains the paragraph beside it, decorates it, or contradicts it.
That hidden relation matters. A student searching “what is a mammal” wants a definition with a good example photo. A shopper wants a product photo whose caption is true. A moderator wants to flag pairs where the picture and the words clash by mistake. Each of these needs a different kind of pair, and a system that only measures “how similar are they?” cannot tell them apart.
The authors call the distance between what a machine sees and what a human means the semantic gap. A photo is a grid of numbers: colours at each pixel. A sentence is a string of symbols. Neither holds its meaning in a form you can compare directly. The gap is also bigger than it looks, because a pair carries a message that neither side carries alone. The paper opens with such a case, a complex message made by a picture plus a few words (its Figure 1).
The popular tools of the time did not close that gap. Image captioning (a system that writes a sentence describing a photo) tries to say what is visible. It does not ask whether a given caption is a good one, a joke, a lie, or an ad slogan with no visible link to the product. The paper says that these approaches “typically address neither semantic interpretations nor the specific role or purpose of an image-text constellation.”
One sentence stays fixed. Pick a picture to put beside it. The class name is the paper’s; the picture is drawn from simple shapes.
Toy example in the spirit of the paper’s bird example for Bad Illustration (Section 3.2). The drawings are ours. The class names and the idea behind them are the paper’s.
Two parrots, two different photos, and the pair is valid both times. That is the first lesson: when the text states a general fact, many pictures fit. The pigeon shares the topic, a bird, but breaks the claim. The car shares nothing. Three different failures, or successes, from one sentence.
The paper makes three contributions, and this lesson follows them in order.
One honesty note up front. This is a 2019 paper, written before large image-text models like CLIP existed. We read it for its ideas: a clean vocabulary for how pictures and words relate, and a careful way to manufacture labels. Chapter 9 says where it sits today.
Chapter 1
Follow the classes from the theorists who studied pictures and words, and watch the list grow from three to eight
Think of three places you meet a picture with words. In a textbook, a paragraph defines “mammal” and a photo of a dolphin sits beside it. In a newspaper, a photo of a crowd has a caption that names the street. In a comic, a speech bubble says “Not again!” over a drawing of a man looking at a clock.
In each place the pair works differently. In the textbook, you could swap the dolphin for a mouse and lose nothing. In the newspaper, the caption tells you what the photo shows, and a wrong caption ruins it. In the comic, neither side makes sense alone, and together they make a joke about being late. The words do not name the clock, and the drawing does not say the man is late.
Before the history, one idea from machine learning. A class is a name we give to a group of examples that behave alike. A classifier is a program that reads an example and answers with one class name. If we want a program to say “this pair is an Illustration”, we first need to agree on the list of class names, and on what each one means. This chapter builds that list.
Researchers in communication science and applied linguistics have studied the “visual/verbal divide” for decades. The paper picks a few of their ideas.
Barthes asked a simple question: which side is more important? He called that the status relation. If the text supports the image, the text is an Anchorage: it pins down what you are looking at. If the image supports the text, the image is an Illustration: it gives an example. If neither outranks the other, the two share the work, and he named that case Relay.
Martinec and Salway extended Barthes. They split the equal-rank case in two: Complementary pairs, where the information in the two sides is intertwined, and Independent pairs, where both sides say the same thing. They also borrowed Halliday’s ideas about how clauses in a sentence connect, and applied them to links between a picture and its text.
McCloud studied comics and noticed a case none of these cover. The picture and the words share no concepts at all, yet they have a strong shared meaning. In his phrase, “pictures and words go hand in hand to convey an idea that neither could convey alone.” He called this Interdependent.
Nöth and van Leeuwen added the dark side: a picture and a text can contradict or contrast. Van Leeuwen notes that an ad may do this on purpose, using opposite colours to draw the eye.
The authors are not communication scientists, and they want something a search engine can use. They had one complaint, borrowed from Bateman: the old taxonomies are so detailed that people disagree about which class a pair belongs to. So they edit the list with four moves.
Step through it. The device adds one thinker at a time and keeps count.
Press Next idea to add each step. Green-ringed boxes are new at this step. A struck-through grey box was removed at this step.
The sequence follows Sections 2.2 and 3 of the paper. The colour of each box shows its source: purple for status classes from Barthes and his heirs, teal for ones the authors added.
The end of the device is the paper’s Figure 3: eight classes, each with a use. Uncorrelated pairs are things to filter out of a result list, or to build an ad blocker. Interdependent pairs are the stuff of ads and marketing search. Complementary pairs suit recommenders, cross-modal retrieval and web search. Illustration and Anchorage matter in educational search, such as textbooks and explanations. The three bad classes are quality checks.
Notice what is not here: a class for pairs with a wrong date or a spelling slip. The classes cover how the two sides relate, not whether each side is individually true. That narrow focus is what makes a computer program for them possible.
Chapter 2
Measure whether a picture and its text share things, then whether the shared things agree
A photo shows a small man in a blue shirt. The text beside it says: “a tall man in a red sweater.” Do the two belong together? Yes, in a way: both are about a man. But a reader who looks at the photo and then reads the words feels a small jolt. Is this even the right photo?
Two separate things happened in that moment. First, the picture and the text shared something: a man. Second, the details they gave about him disagreed. The paper turns these two feelings into two measurements. They come from earlier work by Henning and Ewerth, which the paper cites as [15].
A concept (the paper also says entity) is a thing you can name that appears in the picture or the text: a man, a car, a tree, a diagram. Pictures hold many concepts, from the main subject to a bird far away in the sky. The paper keeps to the ones that matter: “the meaning of an image is related to the main objects in the foreground.” It admits this adds some subjectivity, and says that most of the time the important concepts are easy to find.
Cross-modal mutual information, written CMI, asks one question: how much do the concepts of the picture and the concepts of the text overlap? “Cross-modal” means across the two channels. “Mutual” means shared. Its range is 0 (nothing shared) to 1 (everything shared).
The paper gives no formula, only the idea. To make the idea concrete, we use a toy score of our own: the shared concepts divided by all concepts mentioned by either side. This is a plain ratio of counts, so you can compute it by hand.
Image: man, car, tree. Text: man, car. Shared: man, car (2). Either: man, car, tree (3). So CMItoy = 2 / 3 = 0.67.
Now an important simplification. The earlier work used five levels of CMI. This paper keeps only two, because the goal is to sort pairs into classes, and “overlap or no overlap” is enough. So cmi ∈ {0, 1}: zero means no concept is shared, one means at least some are.
CMI is blind to meaning. If the picture shows a small man in a blue shirt and the text says a tall man in a red sweater, CMI is still positive. The word “man” is in both. Something else must catch the clash.
Semantic correlation, written SC, asks the second question: do the shared things agree? The paper puts it on a line from −1 to +1. At +1 the pair is coherent: both sides tell one consistent story. At 0 they are independent. At −1 they are contradictory: the pair confuses the reader, who may wonder “were they placed together by mistake?”
SC covers more than objects. It includes descriptions, symbols and metaphors, and it often needs knowledge from outside the pair. That is why it is harder to compute than CMI. Again the paper simplifies: sc ∈ {−1, 0, 1}.
Try both measurements on a small scene. The picture is fixed. Pick a sentence and watch the shared concepts link up.
Left: what the toy picture shows. Right: what the sentence mentions. A green line joins a shared concept whose details agree. A red line joins one whose details clash.
Toy scene, our own: a small man in a blue shirt, a green car, a tree. The score is the ratio from the equation above, and the agreement check just compares the listed details. Real SC needs judgement, as the paper says; this device only shows how the two numbers separate.
Look at the second sentence. Two concepts are shared, so CMI is positive. Two of their details clash, so SC is −1. A pair can be high on one measurement and bad on the other. That is why we need both.
Why use two numbers and not one similarity score? Because a similarity score hides the reason. Two pairs can both score 0.4 and mean opposite things: one shares half its concepts and agrees, the other shares half and clashes. The two measurements stay apart, and each becomes a clean training target. Later, a network will predict cmi from 2 outputs and sc from 3 outputs, each a small integer label that the data recipe of Chapter 5 knows how to produce.
Can SC be positive when CMI is zero? Yes. Think of an ad: a picture of a melting alarm clock, with the slogan “Tonight, time is yours.” The slogan does not name the clock, and the clock is not a mug. Yet the pair shares a meaning about relaxing. That is the Interdependent class. Chapter 4 shows which other combinations are possible.
Chapter 3
Add the status dial that separates a definition with a photo from a photo with a caption
A glossary page defines “mammal” and shows a dolphin. A museum label sits under a painting and says who painted it. Both pairs agree with themselves: the shared things match, and nothing clashes. So CMI is 1 and SC is +1 for both. Yet they feel like different things, and a search engine would want to treat them differently.
What differs is who is in charge. In the glossary, the words are in charge, and the dolphin is one example among many. At the museum, the painting is in charge, and the label only helps you read it. CMI and SC cannot see this. They need a third dial.
The paper calls the third dial status, written STAT. It comes from Barthes and appears in “the majority of taxonomies established in the last four decades”, which the authors take as a sign that it is “essential to describe an image-text pair”. Status says how the two sides rank in conveying the overall message. It has three values.
The letter names look backwards at first, so read them as “who is the boss”. In stat = T, the Text is the boss. In stat = I, the Image is the boss. The hero at the top of this page labels the same values “Image is minor” and “Text is minor”.
The paper describes status in words. We can turn its descriptions into a practical test. The test is ours, a rule of thumb built from the paper’s definitions, and not an algorithm from the paper. Ask two questions about any pair.
If only the first swap is harmless, the picture is replaceable, so it is minor: stat = T. The paper says it directly: for Illustration, “the image is replaceable by a very different image without rendering the constellation invalid. If the text is a definition of the term ‘mammal’, it does not matter if the image shows an elephant, a mouse, or a dolphin.”
If only the second swap is harmless, the words are replaceable: stat = I. A caption can be reworded, but the photo cannot change. If neither swap is harmless, both sides hold something the other lacks: stat = 0. And if both swaps were harmless, the two sides would just repeat each other. That is the Independent case the authors dropped in Chapter 1.
Pick a pair. Press Swap the picture, then Swap the words. The scale tilts toward the side that cannot be swapped.
The four pairs are toy examples in the spirit of the paper’s class descriptions. The swap test is our own rule of thumb built from the paper’s wording, not a procedure the authors give. Annotators in the paper judged status by reading the definitions.
Status is the dial people disagree on most, and the authors know it. They quote Bateman’s complaint that the detailed taxonomies of communication science make it hard to put a pair into one class. Their answer is to keep only three values and to give annotators short definitions with one example per class. The swap test above is one way to make those definitions feel concrete. Later, in Chapter 8, you will see how three annotators agreed on the final labels.
Without status, three different classes would sit in one box: Illustration, Anchorage and Complementary all have cmi = 1 and sc = +1. With it, they split into three. The same happens on the clashing side: Contrasting, Bad Illustration and Bad Anchorage all have cmi = 1 and sc = −1, and status splits them in the same way. The negative classes are the same three positions, but with a detail that disagrees. A text about a colourful bird above a grey pigeon is a Bad Illustration because the pigeon is a replaceable example that happens to be the wrong one.
So the full description of a pair is a short triple of small integers, (cmi, sc, stat). Pairs with the same triple get the same class name. That is why the authors say the three metrics “systematically characterize” the classes, and it is the reason Chapter 7 can build a classifier by predicting each of the three separately.
Chapter 4
Count every combination of the three dials, then see why ten of them can never happen
A restaurant prints a menu with 18 dishes. The kitchen can cook eight. For the other ten, an ingredient is missing, or two steps cannot both be done. A sensible chef strikes those ten from the menu before the customers see it.
The three dials of the last two chapters make a menu like that. Overlap has two values. Meaning has three. Status has three. A pair is one choice on each dial, and the number of possible choices is the product.
2 (cmi) × 3 (sc) × 3 (stat) = 18 settings
The paper says it plainly: “If approached naively, there are 2 × 3 × 3 = 18 possible combinations.” A closer look shows that “(only) eight of these classes match with existing taxonomies in communication sciences”, and the remaining ten “cannot occur in practice or do not make sense.” Notice the order of events. The authors did not start from the dials and pick classes. They started from the classes in the literature and checked that the dials describe each one without overlap. That the eight fit neatly into the 18 is the evidence that the dials are the right ones.
| Class | cmi | sc | stat | In plain words |
|---|---|---|---|---|
| Uncorrelated | 0 | 0 | 0 | They do not belong together. |
| Interdependent | 0 | +1 | 0 | No shared things, a joint meaning (ads, comics). |
| Complementary | 1 | +1 | 0 | Each side adds what the other lacks. |
| Illustration | 1 | +1 | T | The text leads, the image is an example. |
| Anchorage | 1 | +1 | I | The image leads, the text says what it is. |
| Contrasting | 1 | −1 | 0 | Complementary, but details clash. |
| Bad Illustration | 1 | −1 | T | An Illustration with the wrong example. |
| Bad Anchorage | 1 | −1 | I | A caption that describes it wrongly. |
This is the paper’s Figure 3 reduced to its core. Read down the cmi column: two classes with 0, six with 1. In the sc column, only the three Bad classes and Contrasting are negative. The status column has 0 for the four classes that do not rank the sides, T for two and I for two.
The paper groups the ten impossible settings into four cases, A to D. Each has a short argument that is worth following, because the same arguments tell the classifier when something is wrong.
3 + 2 + 2 + 3 = 10 impossible → 18 − 10 = 8 classes
Walk one pair down the tree to see how it works. A text describes a colourful bird, and the photo shows a grey pigeon. First question, overlap: a bird is in both, so cmi = 1. Second, meaning: the colour clashes, so sc = −1. Third, status: the text states a general fact and the pigeon is a replaceable example, so stat = T. The three answers meet at one leaf, Bad Illustration. Every pair you can imagine ends at some leaf. The question is only whether that leaf is one of the eight live ones.
Now a pair that should not exist. Suppose an annotator says: nothing is shared, yet the two sides contradict. The tree has no leaf for that. A good annotation guide tells the annotator to look again: either a concept is shared after all, or there is nothing to contradict. In this way the dead leaves act like a spell checker for labels.
Use the device to see where each case falls in the tree. The tree asks its three questions in the order the paper uses for the cascade, which Chapter 7 returns to: first overlap, then meaning, then status.
Each row is one of the 18 settings, read left to right: overlap, meaning, status. Pick a case to light up its dead branches.
The tree is the paper’s Figure 4 redrawn. Branch counts follow Section 3.3: Case A has 3 leaves, B has 2, C has 2 and D has 3.
The ten dead leaves are not only bookkeeping. In Chapter 7 a cascade of three small classifiers predicts cmi, then sc, then stat. If their three answers land on a dead leaf, the system outputs Undefined. That makes the dead leaves a free self-check. On the test set only 27 of 798 pairs (3.4%) ended up there. If the taxonomy were badly drawn, many more would.
Chapter 5
Build 224,856 labelled pairs from old datasets, and make the rare bad classes by swapping a few words
Suppose you must teach a network to tell a Bad Illustration from an Illustration. You need thousands of examples of each, with a human-checked label. You search the web. Good pairs are everywhere. Bad pairs, where a text describes a colourful bird above a grey pigeon, are rare, and nobody has tagged them.
You could pay a crowd to label millions of pairs. The paper rejects that: it is “time-consuming”, and crowdsourced labels need “significant efforts to maintain the quality”. Instead the authors use two strategies. First, collect pairs automatically from existing open sources whose structure already implies the class. Second, augment the data: create new training examples by transforming old ones. Together they produce 224,856 labelled pairs.
Each class gets the cheapest honest source. The trick is to find a dataset whose construction already matches the class definition.
For the last three classes there is “no abundant source”. The authors take a positive pair and damage its text. They replace about 530 keywords (adjectives, directional words and colours) with antonyms or opposites. Their example: “tall man standing in front of a green car” becomes “small woman standing behind a red car”.
The picture keeps its man and its car. The text still names a person and a car. So the overlap stays at 1. But three details are now wrong, so the sc drops to −1. The status stays whatever it was. Turn Complementary into Contrasting, Illustration into Bad Illustration, Anchorage into Bad Anchorage. One pass of word swapping makes three classes. Finally, all texts are cut to 10 sentences.
Pick a caption. Drag the slider to flip more keywords to their opposites. Yellow words were flipped; the small grey word above is the original.
Toy captions and a tiny 20-entry opposites list of our own. The paper uses about 530 keywords from a published list. Its example (tall man, green car) is the first caption; the paper also changes “man” to “woman”. How many words the authors flip per caption is not stated; the slider shows the range.
Move the slider to 1. One word flipped, and the label is already “negative”. Is it really? A reader might shrug at “a tall man” versus “a small man” when the photo is blurry. That is a weakness we return to below.
| Class | Pairs | Where from |
|---|---|---|
| Uncorrelated | 60,000 | MSCOCO, photo and caption mixed |
| Interdependent | 1,007 | Advertisements, hand labelled |
| Complementary | 33,088 | VIST story captions |
| Illustration | 5,447 | ImageNet + Wikipedia summaries |
| Anchorage | 62,637 | VIST single-photo captions |
| Contrasting | 31,368 | Antonym swap of Complementary |
| Bad Illustration | 4,099 | Antonym swap of Illustration |
| Bad Anchorage | 27,210 | Antonym swap of Anchorage |
| Total | 224,856 | Table 1 of the paper |
The set is lopsided. Anchorage has 62,637 pairs and Interdependent has 1,007: a ratio of about 62 to 1. The authors cap the imbalance on purpose (“we have to restrict the level of class imbalance”) and note that they could have made more Anchorage pairs.
Counting by metric (their Table 2) is a handy check. Sum the classes with cmi = 0: 60,000 + 1,007 = 61,007. Sum those with sc = −1: 31,368 + 4,099 + 27,210 = 62,677. Both match the paper. One slip: the paper’s printed status rows put 125,463 on “STAT T” and 9,546 on “STAT 0”. Adding the class counts gives 125,463 for equal status (Uncorrelated, Interdependent, Complementary, Contrasting) and 9,546 for image-subordinate (Illustration plus Bad Illustration). The two labels look swapped in print.
Symptom: the automatic labels do not always agree with people. Measurement: the authors compare them with three human annotators on the 800-pair test set (Table 3, Chapter 8). The recall of Uncorrelated is 69.2%: some “mismatched” pairs were not really unrelated. Bad Illustration has the lowest precision, 69.0%, and annotators often thought those augmented pairs were still valid or uncorrelated. Fix: the paper does not remove the noise. It labels the test set by hand, so the final score is measured against people, not against the automatic labels. That is the honest answer, and it is the reason the test set is small.
Chapter 6
Follow a picture and a text through the system, shape by shape, until they become one prediction
A photo arrives as a grid of 299 by 299 colour dots. A caption arrives as a short string of words. The classifier must say “Anchorage” or “Bad Illustration”. In between, both inputs must become something the same small network can compare. That something is a list of numbers, called a vector.
A vector is an ordered list of numbers, such as [0.2, −1.1, 0.7]. An embedding is a vector that stands for something else, such as a word or a photo, chosen so that similar things get similar vectors. Think of a map: each city gets two numbers, latitude and longitude, and nearby cities get nearby numbers. An embedding is a map with hundreds or thousands of directions instead of two.
Nobody writes the numbers by hand. A network learns them from a large amount of data. A network that was trained on one big task and then reused for another is called pre-trained. The paper reuses two pre-trained encoders, one per modality, so that the new classifier does not have to learn what a dog looks like or what the word “dog” means.
Images are rescaled to 299 × 299 × 3: width, height, and three colour channels. During training they are randomly cropped and their brightness, saturation, hue and contrast are jittered, so the network cannot memorise exact pixels.
The encoder is Inception-ResNet-v2, a deep convolutional network pre-trained on the ImageNet challenge. A convolutional layer slides a small filter across the picture to detect local patterns, such as edges, then textures, then parts of objects. The authors remove the fully connected layers at the end (those are the part that outputs ImageNet class names) and keep the feature maps from the last convolutional layer, with an embedding size of 2048. Averaging over all positions in those maps (average pooling) gives one 2048-number summary for the whole image.
Each word becomes a 300-number vector using fastText, a successor of word2vec. Its special ability is that it works from n-grams: pieces of a word, such as the three-letter chunks “wal”, “alk”, “lki”. A word vector is built from its pieces, so a misspelled or unknown word still lands near its correct form. The paper says this lets the system “recognize a word or derived phrasings despite typing errors”.
A sentence of T words is now a table of shape T × 300. It goes into a bidirectional GRU. A GRU (gated recurrent unit) reads the words one at a time and keeps a running memory, deciding at each step what to keep and what to forget. Bidirectional means two GRUs: one reads left to right, one right to left, and their outputs are joined. The word “bank” then gets context from both sides. The final text embedding has dimension 1024. The paper limits input to 50 words per sentence and 30 sentences per pair.
The paper adds attention. In plain words, attention lets one part of the network look at another part and decide what matters, by giving each item a weight between 0 and 1 that sums to 1. Here, another convolutional layer reduces the image features to 300 dimensions, the same size as a word vector. The authors say this ensures that “the neural network reads the textual information under the consideration of the visual features, which forces it to interpret the features in unison.”
Be careful here. The paper describes this in one sentence plus Figure 7, and gives no formula. We know the sizes (300 on both sides, so a word can be compared with the image) and the purpose, and we should not invent more. If you re-implement it, you must choose a concrete attention form, for example scoring each word against the image vector and weighting the GRU states with those scores.
Concatenate the pooled image vector (2048) and the text embedding (1024) into one vector of 3072 numbers. Four fully connected layers follow, with sizes 1024, 512, 256 and 128. A fully connected layer takes all its input numbers, multiplies by a weight matrix and adds a bias. The last layer, the classification layer, has 2 outputs for CMI, 3 for SC, 3 for STAT, or 8 for the “classic” approach that predicts the class directly.
First dense layer: 3072 × 1024 + 1024 = 3,146,752 numbers.
Trace the data flow with the device. It shows every shape from input to output and counts the new numbers the dense layers add.
Pick which head to train, and drag the number of words. The blue branch is the picture, the warm branch is the text.
Shapes and sizes are the paper’s (Section 4.2 and Figure 7). The parameter counts are our own arithmetic from those sizes, assuming ordinary dense layers with biases fed by the 2048 + 1024 concatenation. The paper reports no parameter counts, and the encoders hold millions more.
Slide the word count. Only the first text shape changes (T × 300). After the GRU, the text is always 1024 numbers, however long the sentence. That fixed size is what lets a 3-word caption and a 300-word story share one classifier.
Why a GRU and not a bag of words? Order matters: “the dog bit the man” is not “the man bit the dog”. Why a pre-trained image encoder? Training one from scratch needs far more pictures than the 224,856 pairs here. The paper does not say whether the encoders are fine-tuned or kept fixed during training; it says only that the Inception layers were pre-trained on ImageNet “to reduce the training effort”.
Chapter 7
Compare one eight-way classifier with three small ones chained together, and see why chaining can lose
You can ask a friend one hard question: “Which of eight kinds is this pair?” Or you can ask three easy ones in turn: “Do they share things? Do they agree? Who leads?” and combine the answers yourself. The paper builds both, trains both, and compares them.
The classic classifier is the network of Chapter 6 with an eight-output head. It outputs one score for each class and picks the largest.
The cascade trains three copies of the same network, with heads of 2 (CMI), 3 (SC) and 3 (STAT) outputs. At prediction time it runs them in a fixed order, CMI ⇒ SC ⇒ STAT, and walks the tree of Chapter 4 with the three answers. The leaf it reaches is the class. If it reaches one of the ten dead leaves, it returns Undefined. The authors argue the cascade is attractive because “it is easier to maintain a good class balance of samples”, and each of the three problems is easier. For example, Contrasting, Bad Illustration and Bad Anchorage all teach the SC network what “negative” looks like. The price is that it “increases the training and evaluation time by a factor of three.”
Set what each small classifier answered. The path walks the tree. Try an answer set that lands on a dead leaf.
The tree and the Undefined rejection are from Section 5.2 and Figure 4. The three input answers are your choices; they are not model outputs.
Training needs a number that says how wrong the network is. The paper uses a softmax cross-entropy loss. Start with the last layer’s raw outputs, called logits: one real number zk per class. The softmax turns them into probabilities that are positive and sum to 1. The loss is the negative log of the probability of the correct class.
A worked toy case with three classes. The logits are z = [2.0, 0.5, −1.0]. Exponentials: 7.389, 1.649, 0.368; their sum is 9.406. So p = [0.786, 0.175, 0.039]. If the true class is the first, L = −ln 0.786 = 0.24, small. If the true class is the second, L = −ln 0.175 = 1.74, large. Training nudges every weight in the direction that lowers the average loss over a batch.
The settings: the Adam optimizer (a method that adapts the step size for each weight) with its default learning rate, dropout of 0.3 on the image embedding and 0.4 on the text embedding (randomly zeroing that share of numbers while training, so the network cannot lean on any single one), batch size 12, 100,000 iterations per model on one NVIDIA Titan X, built in TensorFlow. The split: 800 pairs hand-labelled for testing (100 per class), the rest for training.
Look at the results first. The three single-metric classifiers reach accuracy 90.3% (CMI), 84.6% (SC) and 83.8% (STAT). The cascade reaches 74.3% on the eight classes. The classic classifier reaches 80.8%. The cascade is worse by 6.5 points, although each of its parts is better than 83%. Why?
A cascade needs every stage to be right. If the three stages made independent mistakes, the chance that all three are right is the product of their accuracies. The paper offers its own explanation: “an overall judgment is probably more accurate than the single ones, which only consider one metric.”
Drag each stage accuracy. The yellow bar is what independent errors would give. The lines mark what the paper measured.
The defaults are Table 5 of the paper (90.3, 84.6, 83.8; cascade 74.3; classic 80.8). The product is our arithmetic and assumes the three stages err independently, which real stages do not.
At the defaults, the product is 0.903 × 0.846 × 0.838 = 64.0%. The real cascade scored 74.3%, which is higher than independence predicts. So the errors probably overlap: a hard pair may be hard for all three networks, so the stages fail together instead of separately. This is our reading, not a result the paper reports. Still, 74.3% is below 80.8%, and the paper concludes that “a direct prediction of the image-text class is to be preferred over a combination of three separate classifiers.” That is also practical: you train one network, not three.
The cascade has one prize. Because it can land on a dead leaf, it carries a built-in test of the taxonomy. Only 27 of 798 test pairs (3.4%) did, though 10 of 18 leaves are dead. The classic head cannot offer this check, since it only knows eight classes.
Chapter 8
Read the test set, the confusion matrix and the tables, then list the limits an honest reader should keep in mind
A student scores 80.8% on an exam. Is that good? It depends on who wrote the exam, how hard the questions were, and which questions the student missed. A single accuracy hides all three. This chapter opens the exam: who set it, which questions went wrong, and where the paper’s own numbers disagree.
The authors set aside 800 pairs, 100 from each class, and had three people from their group label them by hand. Each annotator had short definitions of the three metrics, the tree of Figure 4, and one example per class. They could also mark a pair Unsure. The agreement between annotators is measured by Krippendorff’s alpha, a score where 1 means perfect agreement and 0 means no better than chance. Theirs is 0.847. A label counted when the majority agreed. Unsure won the vote for two pairs, which were dropped, leaving 798. The test set is balanced by construction, but human labels moved the final counts (95 Anchorage, 149 Uncorrelated, and so on).
The test labels from people can be compared with the labels the pipeline of Chapter 5 gave the same pairs. That tests the data recipe itself (Table 3).
| Class | Recall | Precision | Test pairs |
|---|---|---|---|
| Uncorrelated | 69.2% | 98.7% | 149 |
| Interdependent | 97.6% | 96.3% | 100 |
| Complementary | 83.8% | 88.0% | 106 |
| Illustration | 83.7% | 80.7% | 95 |
| Anchorage | 90.3% | 87.3% | 95 |
| Contrasting | 89.0% | 78.3% | 87 |
| Bad Illustration | 98.6% | 69.0% | 71 |
| Bad Anchorage | 91.9% | 87.0% | 95 |
Recall is the share of true members of a class that were found. Precision is the share of items given a class that really belong to it. The low recall for Uncorrelated (69.2%) says some of the “mismatched” pairs had a link after all. The low precision of Bad Illustration (69.0%) says that annotators often judged those augmented pairs to be still valid or uncorrelated. So the recipe is good, not perfect, and the weak point is the antonym swap, as Chapter 5 warned.
Table 4 of the paper is a confusion matrix: a grid whose rows are the true classes and whose columns are the predictions of the classic network. A perfect network has numbers only on the diagonal. Every number off the diagonal is a specific mistake. The diagonal sums to 645 of 798 pairs, which is 80.8%.
Rows are the true class, columns are the prediction. Pick a class, then switch between Row (how many true members were found) and Column (how often that prediction is right).
The counts are Table 4 of the paper, assembled from its printed columns. Row totals match the paper’s Sum column, and recall and precision match the paper’s Table 4 percentages.
Three patterns stand out. First, the Uncorrelated row is the weakest: only 67 of 149 are found (recall 45.0%). Most of the rest are called Anchorage (34), Illustration (23) or Bad Illustration (11). The authors blame the image encoder: it may focus on “negligible background elements” and find a link where a human sees none. Second, the same cell pulls Anchorage precision down to 56.5%, since many Uncorrelated pairs land there. Third, mistakes between a class and its negative twin are real but modest: 21 Bad Anchorage pairs called Anchorage, 11 Illustrations called Bad Illustration, 9 Complementary called Contrasting. Positive and negative pairs differ “only in a few keywords, while image content, sentence length, and structure are identical”, so these errors are understandable.
| Classifier | CMI | SC | STAT | Cascade | Classic |
|---|---|---|---|---|---|
| This paper | 90.3% | 84.6% | 83.8% | 74.3% | 80.8% |
| Earlier work [15], mapped to the same levels | 68.8% | 49.6% | not available | not available | not available |
This is Table 5. The earlier method of Henning and Ewerth used five levels of CMI and SC. The authors map those levels onto the two- and three-level scheme (CMI 0, 1, 2 to 0; CMI 3, 4 to 1; SC of plus or minus 0.5 to plus or minus 1) to compare. They beat it by 21.5 points on CMI and 35.0 on SC. Remember that the comparison has a catch: the other method was designed for a finer level scheme and to need little labelled data, so the mapping is only approximate.
Chapter 9
Check your skill on fresh pairs, then keep the numbers, the checklist and the core code
Here is the paper in one paragraph. A picture and its text can relate in eight ways. Three small-integer measurements describe every way: overlap (cmi), agreement (sc) and status (stat). Of 18 possible triples, 8 are real classes and 10 are impossible. To train a classifier without a crowd, the authors collect 224,856 pairs with five collection recipes and make the three bad classes by swapping keywords for opposites. A network encodes the picture with Inception-ResNet-v2 and the text with fastText and a bidirectional GRU, joins them, and predicts the class. Predicting the class directly beats chaining three metric classifiers, 80.8% against 74.3%.
Read the pair. Set the three dials. Press Check. Then press Next pair.
The six pairs are our own toy examples with our own answer key, written from the paper’s class definitions. Real pairs are often less clear, which is why the paper’s annotators reached an agreement of 0.847 and not 1.0.
| What | Number | Why it matters |
|---|---|---|
| Settings of (cmi, sc, stat) | 2 × 3 × 3 = 18, of which 8 real | Ten impossible cases (A to D) are the free quality check |
| Training pairs | 224,856 (Table 1) | Built from MSCOCO, VIST, ads, ImageNet and Wikipedia |
| Largest and smallest class | Anchorage 62,637; Interdependent 1,007 | About 62 to 1 imbalance, kept in check on purpose |
| Keywords swapped for opposites | about 530 | Makes the three bad classes from good ones |
| Test set | 798 pairs, alpha = 0.847 | Hand labelled by three annotators |
| Shapes | image 299 × 299 × 3 → 2048; words T × 300 → 1024; joined 3072 | Dense layers 1024, 512, 256, 128, then 2, 3 or 8 outputs |
| Training | Adam, batch 12, 100,000 iterations, dropout 0.3 and 0.4 | One Titan X, TensorFlow |
| Metric accuracies | CMI 90.3%, SC 84.6%, STAT 83.8% | Earlier work: 68.8% and 49.6% on CMI and SC |
| Class accuracy | classic 80.8%, cascade 74.3% | Errors compound across three stages |
| Undefined leaves reached by the cascade | 27 of 798 (3.4%) | The taxonomy holds together |
This is a sketch in Keras style. It shows the data flow and the label algebra. The attention function is a placeholder, since the paper does not give its formula, and the activation of the dense layers is a choice of ours.
python# 1. The label algebra (Figure 4). Anything else is a dead leaf. CLASS = {(0,0,'0'):'Uncorrelated', (0,1,'0'):'Interdependent', (1,1,'0'):'Complementary', (1,1,'T'):'Illustration', (1,1,'I'):'Anchorage', (1,-1,'0'):'Contrasting', (1,-1,'T'):'Bad Illustration', (1,-1,'I'):'Bad Anchorage'} def to_class(cmi, sc, stat): return CLASS.get((cmi, sc, stat), 'Undefined') # 2. The negatives: same picture, same nouns, flipped details. def make_negative(text, opposites): # ~530 keywords in the paper return ' '.join(opposites.get(w, w) for w in text.split()) # 3. The network (shapes in comments). def build(n_out): # n_out = 2 (CMI), 3 (SC / STAT) or 8 (classic) img = Input((299, 299, 3)) txt = Input((None, 300)) # T fastText vectors fm = InceptionResNetV2(include_top=False)(img) # feature maps, 2048 channels zi = GlobalAveragePooling2D()(fm) # (2048,) key = Conv2D(300, 1)(fm) # image side of the attention, 300-d h = Bidirectional(GRU(512, return_sequences=True))(txt) # (T, 1024); sizes are our split zt = attend(h, key) # (1024,) paper gives no formula x = Concatenate()([zi, zt]) # (3072,) for units in (1024, 512, 256, 128): x = Dense(units, activation='relu')(x) # activation is our choice return Model([img, txt], Dense(n_out)(x)) # logits; train with softmax cross-entropy # 4. The cascade: three models, fixed order, tree lookup. def cascade(pair, m_cmi, m_sc, m_stat): cmi = argmax(m_cmi(pair)); sc = argmax(m_sc(pair)) - 1 # {0,1,2} -> {-1,0,1} stat = 'T0I'[argmax(m_stat(pair))] return to_class(cmi, sc, stat) # may be 'Undefined'
This paper comes from 2019, before models such as CLIP learned one match score from hundreds of millions of image-text pairs. Those models answer “how well do this picture and this text match?” with a single similarity. The paper asks a different question: how do they relate, and is the relation useful or broken? In Lecture 7, Alignment the course covers how modalities are brought into a shared space. The relation classes here show what a single alignment score cannot say: an ad slogan with no shared concept (Interdependent) scores low on matching, and is still a perfect pair.
Its weak points are the ones from Chapter 8: a test set from the same recipe, noisy augmented negatives, and an attention step described in one sentence. Its lasting parts are the vocabulary, the dead-leaf self-check, and the idea of making rare classes by editing common ones.
Credit. This lesson re-teaches “Understanding, Categorizing and Predicting Semantic Image-Text Relations” by Christian Otto, Matthias Springstein, Avishek Anand and Ralph Ewerth (Leibniz Information Centre for Science and Technology TIB, and L3S Research Center, Leibniz Universität Hannover). Read the original on arXiv, or its journal version, “Characterization and classification of semantic image-text relations” (IJMIR 2020). The paper’s data is public at doi.org/10.25835/0010577. The MIT MMAI course is taught by Paul Liang and the MIT Multisensory Intelligence group (schedule). All words, devices and toy examples here are ours.
Now press Present or Teach and explain the three dials from memory: what each measures, why ten of the 18 settings cannot occur, and how the authors made the bad classes. Then go back to the grid at the top, and for each of the eight classes say aloud a pair of your own.