Multimodal Alignment

Teach two encoders to put a picture and its caption in the same place. After 40 training steps all four images find their own caption, and that one pull is how CLIP connected words to the world.

Learn how a model finds the link between a word and a thing it names, from a contrastive loss to optimal transport, time warping and the surprising way alignment appears on its own.

Press Train and watch the four captions swing round the circle to meet their images. Then switch the temperature to Flat and train again: the pairs still meet, but the loss never falls as low. After that we build, piece by piece, everything underneath.

You need a feel for vectors (a list of numbers that points somewhere) and a little probability. We build the rest from zero.

Based on Lecture 4.2 of MIT MMAI: Multimodal AI (Spring 2026), taught by Paul Liang and the MIT Multisensory Intelligence group. Lecture slides (PDF). Every idea is re-taught here in our own words and devices.

Pull the pairs together

0

A toy of the real thing. Each picture and each caption is one direction on a circle, so every vector has length 1, as in CLIP. We run plain gradient descent on the lecture’s InfoNCE loss, in both directions (image to caption and caption to image), over four pairs: blue car, yellow bus, airplane, bowl of cats (the lecture’s own four). The start angles are illustrative. Real CLIP trains two deep encoders on 400 million pairs.

Chapter 0

Finding the Links Between Modalities

See what it means to connect a word to a thing, and sort the four kinds of link a model can learn

Look at a photo. A woman sits on a park bench and reads a newspaper. Under it, a caption says: “A woman reading newspaper.” You do not read the caption and then search the picture. Your eyes jump from the word “woman” to the person, and from the word “newspaper” to the folded sheet in her hands.

That jump is the whole subject of this lesson. A machine has to learn it. Nobody gives the machine a box around the woman. It gets pixels on one side and letters on the other, and it must find out which part goes with which.

The words we need

A modality is one kind of signal: pictures, speech, written text, a heart rate trace, a touch sensor. An element is one piece of a signal: a word, an object in a picture, a half second of sound. Alignment is the job of finding which elements of one modality connect to which elements of another, and how strongly.

The lecture calls this the second core challenge of multimodal AI. It defines the challenge as “identifying and modeling cross-modal connections between all elements of multiple modalities, building from the data structure.” Three sub-challenges follow. Discrete connections link separate elements, like a word and an object. Continuous alignment links signals that flow with no clear elements, like a video and a voice. Contextualized representation lets the links form inside the model with no one marking them.

Today’s session walks through four parts, in this order: what alignment is, explicit alignment with contrastive learning, continuous alignment, and implicit (emergent) alignment. This lesson follows the same order.

Why does a builder care? Because almost every multimodal product leans on a link. Search a photo library with the words “red bicycle”, and the system must link the words to pixels. Ask a robot to “pick up the mug”, and it must link the word to one object on the table. Caption a video, and it must link each sentence to a stretch of time. If the links are wrong, the product fails in ways that look strange: the right words, the wrong object.

Shared and unique: what a link can carry

Two modalities rarely say the same thing. Think of the photo and its caption again. Both tell you there is a woman and a newspaper. Only the photo tells you the colour of her coat. Only the caption tells you, if the writer chose, that the paper is the morning edition.

So each modality holds shared information, which the other one also holds, and unique information, which only it holds. Picture two overlapping circles. The overlap is shared. The two outer crescents are unique. The lecture draws exactly this and names three cases. Stronger overlap means the pair says almost the same thing. Weaker overlap means they share a little. Unconnected means no overlap at all, for example a photo of a beach under a caption about taxes.

This picture will return in almost every chapter. A model can learn only from the overlap, only from the crescents, or from both. Which choice it makes decides what it can do later.

One reading from the session gives these circles names you can measure. Otto and colleagues, in “Characterization and classification of semantic image-text relations”, describe an image and a text with three dials. Cross-modal mutual information asks how much the two overlap. Semantic correlation asks whether the overlap means the same thing in both. Status asks which one leads: does the text explain the picture, or does the picture illustrate the text? The three dials together give eight classes of image and text relation. The paper also shows how to gather a dataset for these classes from the Web, and it trains systems that predict the three dials and the eight classes.

Four kinds of connection

The lecture sorts links into two families, with two kinds in each. The first family is statistical. These links show up in the numbers, whatever the meaning.

The second family is semantic. These links carry meaning.

Sort a few pairs yourself. Pick the kind of link, and watch the overlap picture behind each pair.

Name the link

Read the pair. Choose the kind of link that joins its two sides. The circles show how much the two sides overlap: blue is the picture or signal, warm is the words.

The four kinds and their two families are the lecture’s (slide 6). The five pairs and their overlap amounts are our illustrations, not measurements.

Why start with paired data

The lecture states how most systems learn these links today: with paired data. A pair is two views of one moment, such as a photo and the caption someone wrote for it. The model has no box around the woman. It only knows that this photo and this caption belong together, and that this photo and a caption about a bus do not.

Paired data is cheap, because the web holds an enormous number of photos with text beside them. It is also the first of many limits we meet. A pair says “these belong together” and says nothing about which part goes with which. The model must work that out from thousands of pairs. Chapter 1 shows the machine that learns from pairs. Chapters 2 to 4 show what it learns, and what it misses.

The road ahead follows the lecture:

  1. Aligned representations: two encoders, one similarity function.
  2. Contrastive learning: pull pairs together, push others apart, then CLIP.
  3. What contrastive learning captures: mutual information and its lower bound.
  4. How much to share: the sweet spot between two views.
  5. Representation fission: keep the shared and the unique parts apart.
  6. Global alignment: assignment and optimal transport.
  7. Continuous alignment: warping time, finding change points.
  8. Discretization: turn streams into tokens with HuBERT and VQ-VAE.
  9. Emergent alignment: the Platonic Representation Hypothesis.
  10. When emergence is not enough: bags of words and what to do.
  11. Field guide: numbers, code, links onward.
A photo of a beach sits under a caption about tax law. In the lecture’s picture of shared and unique information, what is the best description?

Chapter 1

Aligned Representations

Build two encoders and one ruler, so that a picture and a caption can be compared as numbers

Imagine two librarians. One shelves books by cover colour. The other shelves them by the first letter of the title. Ask both for “the book about whales” and neither can help the other. They use different shelves, so their positions mean different things.

A photo and a caption have the same problem. A photo is a grid of pixel numbers. A caption is a list of word pieces. You cannot subtract one from the other, and you cannot ask which is closer. To compare them, we first move both onto one common shelf.

Encoders put both onto one shelf

An encoder is a function that turns raw input into a vector, a fixed-length list of numbers. We call that vector a representation of the input. The lecture names the two encoders fA and fB, one for each modality, and the two vectors they produce zA and zB.

Why two encoders and not one? Because the modalities differ in shape and structure. The lecture calls this heterogeneity, and says the first job of the model is to “capture heterogeneity” with specialized encoders. Here is the data flow for a CLIP style pair. These shapes are typical, not fixed.

One function measures the link

The second job is the alignment function g. It takes the two vectors and returns one number that says how strongly they connect. The model learns by tuning three sets of numbers at once: the weights of fA, the weights of fB, and any weights inside g. The lecture writes the training goal as one loss that depends on all three.

g(zA, zB)
The alignment score for one pair. For cosine similarity it lies between −1 and 1.
zA · zB
The dot product: multiply matching entries and add them up. It is large when the two vectors point the same way.
‖zA‖ ‖zB‖
The two vector lengths multiplied. Dividing by them removes length, so only direction counts.

This is cosine similarity, the first of four choices on the slides. Take zA = (3, 4) and zB = (4, 3). The dot product is 3×4 + 4×3 = 24. Each length is 5. So the cosine is 24 / 25 = 0.96, two vectors that point nearly the same way. Now stretch zB to (40, 30). The dot product jumps to 240, but the cosine stays at 0.96. Cosine asks “which way?” and ignores “how far?”

Compare two vectors three ways

The blue arrow zA is fixed at (3, 4). Turn and stretch the warm arrow zB. Watch which scores react to the stretch and which do not.

37°
5.0

The three scores are the exact formulas: cosine, the linear kernel (the plain dot product) and the RBF kernel exp(−‖zA−zB‖² / 2σ²) with σ = 3. The vector zA = (3, 4) is our toy. Bars rescale to fit: the dot product bar is divided by 50.

The other three choices

Kernel similarity (slide 10). A kernel k is any function that scores two vectors and behaves like a dot product in some richer space. The lecture lists four: linear, polynomial, exponential and RBF. The RBF kernel scores by distance: exp(−d² / 2σ²), where d is the gap between the vectors and σ sets how fast the score fades. For (3, 4) and (4, 3), d² = 2, and with σ = 1 the score is exp(−1) = 0.37. Unlike cosine, RBF punishes a stretched vector, because stretching moves it far away.

Canonical correlation analysis, CCA (slide 11). Cosine compares the vectors as they come. CCA first learns a projection U for one side and V for the other, then picks them to maximise the correlation of the projected values. Think of looking at two clouds of points from the one angle where they line up best. Here the encoders fA and fB can be learned too, in a deep version of the method. CCA scores a whole dataset at once, where cosine scores one pair.

Order, hierarchy and pairwise relations (slide 12). Some links are not mirror images. “A dog is an animal” does not mean “an animal is a dog”. A symmetric score such as cosine cannot say which way the arrow points. Methods built for order and hierarchy, such as order embeddings, instead require that a more general thing sits “below” a more specific one, coordinate by coordinate. The slide only names this family; the example is ours.

A failure you can measure: the loud caption

Suppose you pick the plain dot product as g and search a photo library. You type “red bicycle” and the top result is a photo of a tax form. You try “sunset” and the same tax form comes first again. Something is wrong, and it is not the words.

Measure it. Print the length of every image vector. Most lengths sit near 5, but the tax form has length 40. The dot product equals length times length times cosine. So an image with a huge vector scores high against every caption, even when its direction is wrong. Take a caption vector of length 5 at cosine 0.2 with the tax form: the dot product is 5 × 40 × 0.2 = 40. The correct bicycle photo, length 5 at cosine 0.9, scores 5 × 5 × 0.9 = 22.5. The loud vector wins.

The fix is to divide out the lengths, which is exactly cosine similarity. After that, the tax form scores 0.2 and the bicycle scores 0.9. Many real systems go one step further and rescale every vector to length 1 before they compare. Then the dot product and the cosine are the same number. The hero at the top of this page does this: every point sits on a circle.

Where the training signal comes from

Every choice of g needs something to say which pairs should score high. That signal is paired data, as Chapter 0 said. Given paired photos and captions, the loss rewards g when it scores a true pair high, and the gradient (the direction that lowers the loss) flows back through g into both encoders. Both encoders change, so that matching pairs end up near each other. The next chapter makes this concrete.

A caption vector is stretched to ten times its length, with its direction unchanged. Which score stays the same?

Chapter 2

Contrastive Learning and CLIP

Train two encoders with one rule: pull true pairs together and push every other pair apart

You lay four photos on a table and shuffle four captions beside them: “blue car”, “yellow bus”, “airplane”, “bowl of cats”. A child plays a matching game. She does not need anyone to explain what a car is. She only needs to know that each photo has exactly one true caption among the four. After enough rounds she gets good at it.

That game is contrastive learning. The model sees pairs. It must make the true pair score high and the false pairs score low. The word “contrastive” says the model learns by contrast: what a thing is comes from what it is not.

The simplest version: a margin

The lecture starts with positive pairs (a photo and its own caption) and negative pairs (a photo and some other caption). The rule is to bring positives closer and push negatives apart. The first loss on the slide uses a margin α, a gap you demand between the two scores.

L
The loss for one photo with one true caption and one false caption. Zero means the pair is already good enough.
α
The margin: how much higher the true score must be than the false score before the loss reaches zero.
g(zA, zB−)
The score of the photo against a false caption, the negative pair.
g(zA, zB+)
The score of the photo against its true caption, the positive pair.

Here g is a similarity, so higher is closer. (The slide marks the same two terms with colour; if you choose a distance for g, the signs swap.) With α = 0.2, a positive of 0.9 and a negative of 0.5 give max(0, 0.2 + 0.5 − 0.9) = 0. Nothing to fix. A positive of 0.6 against the same negative gives 0.2 + 0.5 − 0.6 = 0.1. The loss is 0.1, so the model nudges the pair.

Early work used this idea for pictures and words. The lecture cites Kiros and colleagues (2014), visual-semantic embeddings: one network for images, one for sentences, one shared space. Their demo does arithmetic in that space: take the vector for a blue car, subtract the word “blue”, add the word “red”, and the nearest pictures show red cars. Words and pictures sit in one region, so the sum makes sense.

The popular version: InfoNCE

The margin loss looks at one negative at a time. The loss behind CLIP looks at every negative in the batch at once. Take one photo and all N captions in the batch. Score them all. Turn the scores into probabilities with a softmax (a function that makes the scores positive and sum to 1, so the biggest score gets the biggest share), and ask: how much probability lands on the true caption? That is a plain classification problem with N classes, where the right answer is “my own caption”.

L
The InfoNCE loss for a batch. Lower is better.
(1/N) ∑i
Average over the N photos in the batch.
esii/τ
The exponentiated score of photo i with its own caption i. The lecture writes this as sim(zAi, zBi), with cosine inside. τ is the temperature.
∑j esij/τ
The same quantity summed over every caption j in the batch, the one true caption and all the negatives.

Work one row by hand. A photo has cosines 0.9, 0.2 and 0.1 with three captions, and the first is true. At τ = 0.1 the scores become 9, 2 and 1. Exponentiate: e9 = 8103, e2 = 7.39, e1 = 2.72. The sum is 8113.2. The true caption gets 8103 / 8113.2 = 0.9988 of the probability, and the loss for this row is −log 0.9988 = 0.0012. Now raise τ to 0.5. The scores become 1.8, 0.4 and 0.2. Exponentiate: 6.05, 1.49, 1.22. The sum is 8.76, and the true caption gets only 0.69. The loss is 0.37. The same cosines, the same photo, a much weaker verdict. That is what temperature does: a low τ sharpens the contest, a high τ flattens it.

Run the contest

Each row is one photo. Each column is one caption. A cell shows the probability that the photo picks that caption. The green ring marks the true caption. Move the temperature and watch the rings fill or empty.

0.07

The sixteen cosine scores are our illustration, chosen so that row 4 has a hard negative: caption 2 (0.52) is nearly as close to photo 4 as its own caption (0.55). The loss is the exact InfoNCE formula above, image to caption only.

CLIP: the same contest at web scale

CLIP (Radford and colleagues, 2021, “Learning Transferable Visual Models From Natural Language Supervision”) runs this contest on a huge scale. It trains an image encoder and a text encoder on 400 million image and text pairs from the web. Each training batch holds 32,768 pairs. The batch gives a 32,768 × 32,768 grid of scores, about 1.07 billion cells. Only the diagonal holds true pairs. Every other cell is a free negative, so one batch teaches the model from over a billion contrasts for the cost of 65,536 encoder passes. CLIP also learns the temperature: it starts at 0.07. The loss runs in both directions, photo to caption and caption to photo, and the two results are averaged. Our CLIP lesson builds the full model.

The payoff is that the encoders transfer. After pretraining, you can classify a photo with no new training: write a caption for each class (“a photo of a dog”, “a photo of a cat”) and pick the nearest. The lecture’s remark is the one to hold on to: CLIP’s two spaces, zL and zV, are coordinated but not identical. They are two separate spaces that agree on distances for true pairs.

Contrast has a failure mode too. Suppose two photos in one batch both show a dog, and both have the caption “a dog”. The loss still treats photo 2 as a negative for photo 1’s caption and pushes them apart. The symptom is a model that separates near duplicates it should merge. You can measure it by counting batch pairs whose captions match, then check how often they fall in one batch. The usual fixes are to deduplicate captions or to use a huge, shuffled batch so clashes stay rare.

You raise the temperature τ from 0.1 to 0.5 and leave every cosine score unchanged. What happens to the probability on the true caption?

Chapter 3

What Contrastive Learning Actually Captures

Measure how much a photo tells you about its caption, and see why a contrastive loss can only count so high

A friend hands you a photo and keeps its caption hidden. Before you see the caption, you guess what it says. Then you read it. If the guess was already close, the caption taught you little. If the caption surprised you, it taught you a lot. How much a photo reduces your surprise about its caption is exactly what we want to measure.

The lecture claims that contrastive learning, the machine of Chapter 2, secretly measures this. It captures the information the two sides share, and little else. This chapter shows why.

Mutual information, from a tiny example

Mutual information, written I(X; Y), is the number that says how much knowing one variable tells you about another. It is zero when they are independent. It grows as they depend on each other. The slide writes it as an average of a log ratio.

I(X; Y)
Mutual information, in nats if the log is natural. Zero means the two say nothing about each other.
EX,Y
The average over every pair (x, y), weighted by how often that pair really occurs.
PXY(x, y)
How often x and y occur together, as they truly do in the data.
PX(x) PY(y)
How often they would occur together if they were independent: just the two separate chances multiplied.

Try it on a toy. Let a photo show a cat (X = 1) or a dog (X = 0). Let its caption say “cat” (Y = 1) or “dog” (Y = 0). Suppose captions are right 80% of the time, and photos are half cats, half dogs. The joint chances are 0.4 for cat and cat, 0.4 for dog and dog, 0.1 for each mismatch. Each separate chance is 0.5, so independence would give 0.25 for every cell. The ratio is 0.4 / 0.25 = 1.6 on the matching cells and 0.1 / 0.25 = 0.4 on the mismatching cells. Now average the logs: 0.8 × ln 1.6 + 0.2 × ln 0.4 = 0.8 × 0.470 − 0.2 × 0.916 = 0.376 − 0.183 = 0.193 nats. A perfect caption would score 0.693 nats, the full one bit of the cat or dog choice. A random caption would score 0.

Why a contrastive loss counts mutual information

Slide 20 gives the argument in three steps. We walk them in plain words.

Step 1: the loss uses a critic. In InfoNCE, a critic function f(xA, xB) scores a pair. In CLIP, the critic is exp(cos / τ) from the last chapter. Training turns the critic into a detective. Real pairs come from the true joint, p(xA, xB). Shuffled pairs come from the product of separate chances, p(xA) p(xB). The critic must tell them apart, so it acts as a binary classifier on “real or shuffled?”.

Step 2: the best critic is a ratio. The lecture states that at the optimal loss, the critic equals f*(xA, xB) = p(xA, xB) / ( p(xA) p(xB) ). That is exactly the ratio inside the mutual information. It is large for pairs that go together more often than chance, and small for pairs that go together less often.

Step 3: plug it back in. Put f* into the loss. The result, from the original contrastive predictive coding paper (van den Oord and colleagues, 2018), is an inequality:

I(XA; XB)
The true mutual information between the two modalities.
log N
The log of the batch size: how many captions the photo must pick from.
L*
The loss the best critic can reach. A lower loss gives a higher, tighter bound.

In words: training lowers the loss, and a lower loss proves more mutual information. Contrastive learning “captures” mutual information, because it optimises a lower bound on it. Pairing the loss with the lecture’s Venn picture, CLIP focuses on the shared region of the two circles. Whatever lives only in the photo (the exact brush strokes) or only in the caption (the writer’s tone) pays no reward.

The ceiling: you cannot count past log N

The loss cannot be negative. So L* ≥ 0, and the bound can never exceed log N. A batch of 64 gives at most ln 64 = 4.16 nats. A batch of 32,768, CLIP’s batch, gives at most ln 32,768 = 10.40 nats, which is 15 bits. If the true mutual information is larger, the bound reads the ceiling and stops. The symptom is a loss that falls close to zero and then flattens, even though the encoders could still improve. The measurement: compare the final loss to zero, and compare log N to the dataset’s plausible information. The fix is a larger batch, or an estimator that does not saturate.

Read the bound

The gray bar is the ceiling log N for your batch size. The purple bar is the true information. The teal bar is the most a perfect critic can prove. Drag both sliders and watch the teal bar hit the ceiling.

64
6.0 nats

The ceiling log N follows from the lecture’s inequality and L* ≥ 0. The teal bar uses min(true information, log N), the best a perfect critic can show; the true information is a value you choose, not a measurement. N runs from 4 to 32,768 (CLIP’s batch).

From coordination to fusion

The lecture ends this part with a use of the same idea outside alignment. Colombo and colleagues (2021) add a mutual information term, LMI, to a fusion model’s task loss Ldown. The extra term pushes the fused vector z to carry what the two modalities share. The slide asks you to name the assumption: information present in both modalities is the most important for the downstream task. That is not always true. A sarcasm label might live in the tone of voice that the transcript hides. The next chapter asks how much sharing is right.

A team trains with a batch of 256 and watches the loss reach 0.02. Their bound on mutual information is log 256 − 0.02 = 5.52 nats. What can they conclude?

Chapter 4

How Much Should Two Views Share?

Find the sweet spot between two views that share too little to teach and too much to generalise

Two security cameras watch one room. The task: say whether anyone is home. If both cameras sit side by side and show nearly the same frame, a model learns to match them and picks up every speck of dust along the way. If one camera faces the wall and the other the ceiling, the two frames share almost nothing, and the model has no signal to match. Somewhere between lies a pair of cameras that agree on exactly the facts that matter.

That is the question on slide 21: how much information should be shared? The slide shows three overlaps. One says “not enough signal”. One says “just right”. One says “too much noise”.

Views, and the assumption behind them

In contrastive learning, a view is one version of an example that you feed to an encoder. For an image, two views can be two random crops, or the same photo in two colour spaces. For a multimodal pair, the two views are the two modalities. The model learns to match the views of one example and to separate views of different examples, as in Chapter 2. What it learns is what the views have in common (Chapter 3).

The classic argument about when this works relies on multi-view redundancy. It says that each view alone already holds everything needed for the task: I(X1; X2) = I(X1; Y), so what the views share is exactly what the label needs. Tosh and colleagues (2021) study contrastive learning and linear predictors under this kind of assumption. The lecture’s warning, in the bottom caption of the slide, is blunt: multi-view redundancy may not hold for multimodal problems. A photo shows a face. A voice carries emotion. Neither alone holds the whole answer.

The reading: what makes a good view

Tian, Sun, Poole, Krishnan, Schmid and Isola (NeurIPS 2020) ask the question directly in “What Makes for Good Views for Contrastive Learning?” (our paper lesson: What Makes for Good Views). Their answer is the InfoMin principle: a good pair of views shares as little information as possible while keeping everything the task needs. In symbols, minimise I(v1; v2) subject to both views keeping the task information, I(v1; y) = I(v2; y) = I(x; y).

The paper names three regimes, which are the three pictures on the slide:

So performance should follow a reverse-U as sharing grows: it climbs while views add task information, peaks at the sweet spot, and then falls. The authors test this by turning one knob, data augmentation (random changes to a photo such as cropping and colour jitter). Stronger augmentation makes two views of one photo more different. They find that stronger augmentation lowers the information the views share, and that accuracy improves, up to a point. Their best recipe reaches 73.0% top-1 accuracy on the ImageNet linear readout (a simple classifier trained on frozen features) with a ResNet-50, a state of the art for unsupervised pretraining at the time.

Tune the views

Choose a task. Then turn up the augmentation. The two circles are the two views; the green disc is what the task needs. Find the strength where the overlap just covers the green disc.

2.0

A cartoon of the paper’s reverse-U, not its data. We set the shared information to 8 minus 0.7 times the strength (in nats), the task needs 1.5, 3 or 5 nats, and accuracy rises linearly to the sweet spot and then falls gently. The three regimes and the shape are the paper’s; every number here is illustrative.

Work the device by hand. The breed task needs 3 nats. At strength 2, the views share 8 − 0.7 × 2 = 6.6 nats. That is 3.6 nats more than the task needs, so the model also learns 3.6 nats of background, lighting and pose. At strength 7, the views share 8 − 4.9 = 3.1 nats, almost exactly the task. At strength 10, they share 1.0 nats, 2 less than needed, so the model cannot tell two breeds apart. The same strength is right for one task and wrong for another: the “day or night” task peaks near strength 9, the “read the sign” task near strength 4.

A failure you can measure

Picture a run (the numbers are illustrative). You pretrain on photos with only a light crop. A linear probe (a one layer classifier trained on frozen features) on a breed task reaches 60%. You add stronger colour jitter and cropping, and it climbs to 68%. You add even more, and it falls to 63%. The symptom is a probe score that rises, then drops, as you turn one knob. The measurement is a sweep: run the same probe at five augmentation strengths and plot the curve. The fix is to pick the peak. The paper goes further and learns the views instead of guessing them, with a generator network that makes a second view while a loss keeps the shared information low.

For multimodal data you do not get to pick augmentations. The two modalities are what they are, and their overlap is fixed by the world. So the lesson becomes a different warning: if you train only on what the modalities share, you throw away what each holds alone. The next chapter keeps both.

Two views share far more information than the task needs, including irrelevant detail. According to the InfoMin picture, what happens?

Chapter 5

Representation Fission: Keep Shared and Unique Apart

Split what is shared from what is unique, so that a model can use the facts only one modality holds

A cartoon in a magazine shows a crowded dinner table. The caption reads: “Can you please pass the cow?” The joke needs both parts. The caption alone is a polite request. The drawing alone is a dinner. Together they are absurd. Yet the funny part lives in neither side by itself: it lives in the way they clash.

Chapters 2 to 4 trained a model on what both sides share. For this cartoon that is thin: both mention a table and a request. A model that keeps only the shared part has thrown away the cow.

Fission: split one pile into labelled piles

The lecture names the fix representation fission. Its definition: learn a new set of representations that reflects the internal structure of the multimodal data, such as how it factorises or clusters. The word “fission” means splitting, and the picture is a tree: one input splits into branches. The slide shows two kinds.

Slide 23 lists what falls in each part for a photo and its words.

Look back at the Venn picture of Chapter 0. The middle region is the shared part that CLIP captures. Fission names the two crescents too, so a model can keep them.

Two rules for factorised learning

Tsai and colleagues wrote a well known method that learns factors like these (ICLR 2019; the slide lists 2021). The slide reduces it to two goals, with the shared region marked as z.

Goal 1: maximise mutual information. Each factor should carry a lot about the modality it describes. In the notation of the slide, make I(z; modality) large, once per modality.

Goal 2: minimise conditional entropy. Entropy measures how uncertain a variable is. Conditional entropy, written H(z | A), measures how uncertain z stays once you know modality A. If it is zero, A determines z completely. So this goal says each factor should be settled by the modality it belongs to, and carry nothing that came from somewhere else. Together, the two goals make a factor that is rich and clean.

Factorized contrastive learning

Liang and colleagues (NeurIPS 2023), in “Factorized Contrastive Learning: Going Beyond Multi-view Redundancy”, apply the same idea to the contrastive machine of Chapter 2. A normal contrastive model uses only cross-modal contrast: pull a photo and its own caption together. That finds the shared part. FactorCL adds self contrast inside each modality: pull two altered versions of the same input together. The slide’s example is the caption: “Can you please pass the cow?” and “Can you kindly pass the cow?” are two views of one caption. They differ in wording, and the contrast teaches the model what this modality alone knows. The paper’s title promises the result: it goes beyond the multi-view redundancy of Chapter 4. The slide’s conclusion is that it learns both shared and unique information. (The full method also adds terms to keep the unique parts relevant to the task. We do not derive them here.)

Now try the effect with data you control. The toy below builds each example from three hidden bits. Bit S is in both modalities. Bit UA is in modality A only. Bit UB is in B only. You pick a task, and an objective. The device then trains a lookup classifier on the lanes the objective keeps, and tests it on fresh examples.

Keep the right bits

Pick what the task needs. Then pick what the model is trained to keep. Lanes that pass the filter are bright. The test accuracy comes from a classifier trained on only those lanes.

A toy, built so that the answer can be checked by hand. Each example has three independent fair bits. The classifier is a majority vote on the kept bits, trained on 400 seeded examples and tested on 400 others. Cross-modal contrast keeps only what both modalities hold; adding self contrast keeps each modality’s own bit too. This is our model of the effect, not the paper’s experiment.

Run the grid. With cross-modal only, the first task is solved at 100% and the other three sit near 50%, which is a coin flip. The fourth task, shared and A’s own, is the cartoon: the label is the answer of one bit set against another, and no single lane predicts it. Add self contrast and every task reaches 100%.

A failure you can measure

A team builds an image and text search with CLIP style training, and asks it to find “glossy” versus “matte” product photos. Accuracy stays near 50%. The symptom: a task that depends on texture, which captions rarely mention, fails even though retrieval looks fine. The measurement: probe the frozen image vectors for texture. If a linear probe cannot read it, the training objective threw the texture away. The fix: add an objective that keeps what the image alone holds, such as self contrast between two augmentations of one photo.

A task needs a fact that only the image holds. A model trained with cross-modal contrast alone is tested on it. What do you expect?

Chapter 6

Global Alignment: Assignment and Optimal Transport

Match every element to a partner at once, and then let each element split itself across several partners

Four dancers stand on one side of a hall. Four dancers stand on the other. A coach has a score for every cross pairing: how well Ana dances with Ben, Ana with Cy, and so on. Each dancer can have exactly one partner. If the coach lets every dancer simply pick the highest score, two dancers may choose the same partner, and someone is left alone. The coach needs a global plan.

Slide 26 sets the same scene for words and pictures. The pairing between elements, such as which word goes with which object, may be hidden: the slide marks it with a question mark and calls it latent pairing information. A model should then learn two things at once, the representation (what each element looks like as a vector) and the global alignment (which element pairs with which). The coordination function g of Chapter 1 supplies the score for each pair. The global alignment step picks a whole pairing from those scores.

The assignment problem

Start with the strict case. The slide gives three assumptions: both modalities have the same number of elements, the alignment is one to one (a “hard” alignment), and every element is assigned. The last two together make a perfect matching in a bipartite graph, a graph whose edges only join the two sides.

Write the weight of pairing element i of A with element j of B as wij = g(zAi, zBj). An assignment is a function f that maps each i to one j with no repeats, in other words a permutation. The goal is to maximise the total weight.

maxf
Search over every assignment f, every way to give each A element its own B element.
∑i=1N
Add up the weights of the N chosen pairs.
wi, f(i)
The score of pairing element i with the partner f gives it.
xij
A switch: 1 when i and j are matched, 0 otherwise. The right hand side is the same problem written with switches.

How do you solve it? The naive way checks every assignment. There are N! of them. For 4 elements that is 24. For 10 it is 3,628,800. For 20 it is about 2.4 × 1018. That is too many. The lecture’s better solution is linear programming: write the problem with the switches xij, allow them to take any value between 0 and 1, and solve with the simplex algorithm (a classic method that walks along the edges of the allowed region to its best corner). For assignment, the best answer of the relaxed problem lands on whole 0 or 1 switches, so you get a true matching. (The Hungarian algorithm solves it directly in about N3 steps; for 20 elements that is 8,000 steps.)

Optimal transport: soft matches

Real data breaks the three assumptions. A sentence has 9 words and the picture holds 4 objects. One word, “people”, may point at three faces. The slide drops the assumptions: different numbers of elements, and many to many “soft” alignment. Now xij is not a switch but an amount, how much of i is linked to j.

The picture is transport. Each A element holds some mass, say a pile of sand. Each B element has a hole of some size. You move sand from piles to holes, and every pile must be emptied while every hole is filled. Moving sand between a poorly matched pair costs more than between a well matched pair. The cheapest plan is the optimal transport plan, and its cost is the Wasserstein distance that the slide names. In the lecture’s similarity form, the plan maximises the total weight, ∑ wij xij, with the constraint that rows and columns of x add up to the masses. (The slide shows the objective only; the row and column sums are the standard constraint.)

A popular way to compute a soft plan is Sinkhorn’s algorithm. Turn the weights into a positive table Kij = exp(wij / ε). Then repeat two steps: scale every row so it sums to its mass, then scale every column so it sums to its mass. A few dozen rounds give the plan. The knob ε sets how soft the plan is. A small ε makes a nearly hard matching. A large ε spreads each element thinly over all partners.

Match them all at once

Rows are images, columns are captions, and each cell holds a similarity. Pick Greedy to let every row choose alone, Assignment for the best one to one matching, or Soft plan for optimal transport. Press New scores to try other tables.

0.10

The scores are seeded toy numbers between 0 and 1. Greedy takes each row’s best cell. Assignment checks all 24 one to one matchings and keeps the best total. The soft plan runs 60 rounds of Sinkhorn with equal masses of 1/4 on each side. The softness slider only affects the soft plan.

A failure you can measure

A system matches 4 words to 4 image regions by letting each word take its best region. The symptom: two words, “man” and “person”, both grab the same region, and the region holding the newspaper is never chosen. The measurement is easy: count how many regions are chosen more than once. In the device, greedy often scores this collision above zero. The fix is to demand a global plan, with assignment when you know the counts match, or with a soft plan when they do not. Both are tied to a total score, so one element cannot hog a partner for free.

A caption has 9 words and the image has 4 detected objects. Several words should link to one object. Which tool fits?

Chapter 7

Continuous Alignment: Warping Time

Align two signals that flow with no clear elements, by stretching one in time until it fits the other

Two dancers learn the same routine. One moves fast through the opening and slows for the finale. The other does the reverse. You film both and want to compare move by move. Frame 100 of the first video shows the spin. The spin in the second video happens at frame 160. If you line the videos up by frame number, you compare a spin with a bow.

So far every element had a clear edge: a word, an object, a photo. Here there are no edges. The input is a signal that flows, such as a video, a voice or a sensor trace. The lecture defines continuous alignment as modelling the alignment between modalities with continuous signals and no explicit elements. It names two ways to get there: continuous warping (this chapter) and discretization, also called segmentation (the next).

Dynamic time warping, from a tiny example

Warping means stretching and squeezing time. Dynamic time warping (DTW) finds the stretch that makes two sequences match best. It builds a table. Rows are the time steps of sequence A, columns those of B. Each cell holds the cost of matching that A step to that B step, such as the absolute difference of their values. A path through the table, from the top left to the bottom right, is one alignment. The rule: the path only moves right, down, or diagonally. Time never runs backward, and no step is skipped.

D(i, j)
The cheapest total cost of any path that ends by matching step i of A to step j of B.
|ai − bj|
The cost of this one match: how far apart the two values are.
min{ … }
Take the cheapest of the three cells you could have come from: above, to the left, or diagonal.

Work it on four numbers each. Let A = (1, 2, 3, 3) and B = (1, 1, 2, 3). The match costs are 0 for equal values and grow with the gap. Fill the table row by row. Row 1 reads 0, 0, 1, 3. Row 2 reads 1, 1, 0, 1. Row 3 reads 3, 3, 1, 0. Row 4 reads 5, 5, 2, 0. The last cell is 0, so the best path costs nothing. Trace it back and you get these matches: A1 with B1 and B2, A2 with B3, A3 with B4, and A4 with B4. One A step matches two B steps, and one B step matches two A steps. That is the stretch. Straight index matching, A1 with B1, A2 with B2, and so on, would cost |2−1| + |3−2| = 2.

The table has N × M cells, and each cell costs one lookup, so DTW takes about N × M steps. That is why it still runs on long signals.

Canonical time warping: different sensors

DTW compares values directly, so both signals must live in the same units. A video of a dancer and a motion capture suit do not. Zhou and De la Torre (NeurIPS 2009) join DTW to CCA, the projection method from Chapter 1, in canonical time warping. It takes turns between two steps: find the projection that makes the two signals most correlated, then find the warp that best matches the projected signals. The lecture’s slide shows it aligning video sequences of human behaviour.

Warp, then find the breaks

In Warp mode, the lower signal is the upper one played at an uneven speed. Raise the strength and watch the straight lines of the match bend. In Change points mode, find the three jumps in a noisy signal by moving the threshold.

0.60

Toy signals made in the page. The lower signal is the upper one resampled through a smooth, uneven time map, so the true match is known. DTW is the exact table above with absolute difference cost. In Change points mode the score at each step is the gap between the mean of the next 10 samples and the previous 10, on a signal with true jumps at samples 30, 60 and 90 and seeded noise.

Change points: where the signal changes its mind

Sometimes you do not want to match two signals. You want to cut one into regions. Changepoint detection (slide 31) finds the moments where a signal’s behaviour shifts: a heart rate that jumps when a person starts to run, or a voice that changes speaker. The simplest detector slides a window. At each time it compares the average just before with the average just after. A big gap marks a change. In the device, a low threshold sees the noise as jumps. A high threshold misses the small change. The right threshold sits between the noise gap and the smallest real jump.

The cuts it finds turn a flowing signal into pieces. Those pieces are elements, and once you have elements, the discrete machinery of the earlier chapters applies. That is the bridge to the next chapter.

A failure you can measure

A team aligns a lecture’s slides with its audio by index: slide 1 at 0 to 2 minutes, slide 2 at 2 to 4, and so on. The symptom: the audio talks about the third slide while the screen shows the second, and the error grows through the hour. The measurement: pick five spots, mark the true slide change by hand, and measure the gap in seconds. The fix is a warp: score each audio window against each slide and let DTW find the monotone path through the table.

In dynamic time warping, why can one step of A match two steps of B?

Chapter 8

Discretization: Turn Streams into Tokens

Cut a continuous signal into units, then give each unit a name from a fixed list

You say “tomato” into a voice assistant. The microphone records a smooth wave, about 0.6 seconds long. There are no gaps between sounds, no spaces, no letters. Yet the word has six sounds: t, ah, m, aa, t, ow. To read the wave as words, a system must first decide where one sound ends and the next begins.

This is the second route to continuous alignment, called discretization or segmentation. Instead of warping time, we cut the signal into pieces that behave like elements. The lecture lists the assumption it makes: segmented elements. Its three examples are medical images split into regions, signals cut into events, and photos split into objects.

The speech example: many frames, one phoneme

A phoneme is one basic sound of a language. A spectrogram turns the wave into a picture: time runs across, pitch runs up, and brightness shows energy. It is cut into frames, short slices of time. A common frame step for speech models is 20 milliseconds, which gives 50 frames per second. So “tomato” at 0.6 seconds has about 30 frames for 6 phonemes, roughly 5 frames each.

The slide asks: how can we predict the sequence of phoneme labels? The difficulty on the next slide is many to one alignment: many frames map to a single phoneme, and nobody tells the model where the borders lie. The model sees t t t t t ah ah ah ah m m m … and must output the six labels once each. Typical remedies include collapsing repeated labels, or letting the model learn the borders from the sound itself, which is what the next method does.

HuBERT: let clustering invent the labels

Labelled speech is expensive. HuBERT (Hsu and colleagues, 2021, Hidden-unit BERT) avoids labels. It follows a loop that the slide draws with an orange arrow.

  1. Cluster. Run k-means clustering on speech features. K-means picks K centre points, and assigns every frame to its nearest centre. Each frame now has a pseudo label: a cluster number c, called a hidden unit.
  2. Mask. Hide some of the input frames (the red crosses on the slide).
  3. Predict. A self-attention transformer reads the damaged sequence and must predict the cluster number of each masked frame. Its outputs pass through a linear layer to class scores.
  4. Repeat. Take the features the transformer learned, cluster them again, and train again. The labels get better each round.

In the paper’s first round, for example, the clusters come from simple audio features. The clusters need not match phonemes. They only need to be consistent: the same sound should land in the same cluster, so predicting it from context teaches the model about sound structure.

VQ-VAE: a visual alphabet

Images can be discretised too. A VQ-VAE (van den Oord and colleagues, 2017, “Neural Discrete Representation Learning”) has an image encoder, a decoder, and between them a codebook: a list of vectors, one per allowed token. The encoder turns each patch of the image into a vector. The system replaces that vector by the nearest codebook vector, and passes only the index of the winner on. The slide’s picture shows a cat photo becoming [98] [3] [990], a short list of whole numbers in the range 0 to 8192. Each number is a visual token. The decoder rebuilds the picture from those tokens.

Here is the lookup on a toy. Let the codebook hold four vectors: (0, 0), (1, 0), (0, 1), (1, 1), numbered 0 to 3. An encoder output of (0.8, 0.3) has squared distances 0.73, 0.13, 1.13 and 0.53 to the four. The smallest is 0.13, so the token is 1, and the quantisation error is the distance to (1, 0), about 0.36. The error is the price of the cut.

Cut the stream into tokens

Each dot is a frame of a signal, with two features. The path runs from frame 1 to 36. The big rings are the codebook, found by k-means. Raise K and watch the tokens below the plot change.

3

A toy. The 36 frames are seeded points that pass through three blobs and return to the first, so the true number of sounds is 3 and the path has 4 stretches. The codebook is plain k-means with farthest point starts, run to convergence. Real HuBERT clusters 39 dimensional audio features (or learned features) into 100 or more clusters.

A failure you can measure

You train a VQ-VAE with a codebook of 8,192 entries. The reconstructions come back blurry, as if the decoder only ever sees a handful of colours. The symptom is a blurry decoder output. The measurement is a histogram of how often each code is chosen: if 400 of the 8,192 get nearly every hit, the rest are dead codes, a failure usually called codebook collapse. The fix is to restart dead codes at random encoder outputs, or to update the codebook with moving averages. The same check applies to HuBERT clusters: if one cluster takes 60% of the frames, the targets carry little to predict.

HuBERT never sees phoneme labels. Where does its training target for a masked frame come from?

Chapter 9

Emergent Alignment and the Platonic Hypothesis

Watch two models that never saw each other’s data come to agree on which things are alike

Two strangers each describe the same fruit bowl. One has only photos and has never read a word. The other has only text and has never seen a picture. Ask each: is an apple more like an orange, or more like an elephant? Both say orange. Neither was told to agree.

Chapters 1 to 8 forced alignment on purpose. We gave the model pairs, a loss and a similarity function. Slide 37 shows another route, called implicit or emergent alignment: models trained apart, with no cross-modal loss, still end up with matching structure. The slide sets “explicit alignment” (two encoders joined by a link) next to “implicit alignment”, where the box at the right holds a quote.

The hypothesis

The quote is from Huh, Cheung, Wang and Isola, “The Platonic Representation Hypothesis” (ICML 2024). It says: neural networks, trained with different objectives on different data and modalities, are converging to a shared statistical model of reality in their representation spaces. The name comes from Plato’s cave. The prisoners see shadows on a wall. Images and text are two different kinds of shadow cast by one real world. A model good enough at either kind has to rebuild some of the world behind it, and two such rebuilds look alike. Our Platonic Representation Hypothesis lesson takes this paper apart in full.

The paper gives three pressures that push models together. Task generality: a model trained for many tasks has fewer ways to be right. Model capacity: a bigger model can reach the best solution. Simplicity bias: among many solutions, training prefers the simple ones, and the simple solutions are shared.

How to compare two spaces

There is a puzzle. The image model gives 768 numbers per picture. The text model gives 4,096 numbers per word. You cannot subtract them or take a dot product between the two. So how do you ask whether they agree?

The answer on slide 40 is to compare relationships, not coordinates. Build a kernel for each model: a table that holds, for every pair of items, how similar the model finds them.

Kimg(i, j)
How alike the image model finds item i and item j: the dot product of their image vectors.
Ktext(i, j)
The same for the text model, using the words yi and yj that name the same two items.
Kimg ≈ Ktext
The two tables agree for all pairs. The vectors may live in different spaces, but the pattern of “who is near whom” matches.

Worked example with the slide’s three items. The image model scores apple and orange at 0.8, and apple and elephant at 0.1. The text model scores “apple” and “orange” at 0.7, and “apple” and “elephant” at 0.15. The numbers differ, the vector lengths differ, but both tables put orange near apple and elephant far away. The tables agree.

The paper measures agreement with mutual nearest neighbour alignment. For each item, take its k nearest neighbours in the first space and in the second space. Count how many are the same. Average over all items. A score of 1 means every item has the same neighbours in both models. The chance level is about k / n, for n items. With k = 10 and n = 10,000, chance is 0.001.

Compare the neighbours

Left: nine items in an image model’s space. Right: the same nine in a text model’s space, which is turned and stretched. Pick an item to see its two nearest neighbours in each space. Raise the model scale and watch the neighbours agree.

apple
1

A cartoon of slides 38 to 40, not their data. Three groups (fruit, animals, vehicles) sit in a hidden shared layout. Each model sees it with noise; the text space is also rotated 50 degrees and stretched, and we make the text noise shrink as the scale grows. Alignment is the mutual 2-nearest-neighbour overlap over all nine items; chance is 2/8 = 0.25.

The evidence on the slides

Slide 38 plots language models against vision models. The horizontal axis is language performance. The vertical axis is how well the language model’s space aligns with DINOv2, a vision model. Each point is one language model, and the points climb: the stronger the language model, the more its space resembles the vision model’s. Slide 39 turns the picture round. It plots a language task score (HellaSwag, a commonsense test; GSM8K, a maths test) against alignment to vision. Better aligned models score higher. The vision model here, DINOv2, is a successor of the self-supervised model in our DINO lesson; its features were learned from images alone, with no captions. The agreement with a model trained on text alone is the surprise.

A caution before you lean on it. These are trends across many models, and the paper itself calls it a hypothesis. The next chapter asks what happens when the trend does not hold.

Here is how to read such a score without fooling yourself. Suppose a report says two models reach alignment 0.15. That sounds small. Check the chance level first: with 10,000 items and k = 10, chance is 0.001, so 0.15 is 150 times chance. Then read it the other way: 85% of each item’s neighbours still differ between the two models. Both readings are true. The symptom of a careless claim is a score quoted with no chance level, or measured on 50 items where chance is already high. The measurement is to report the score, the chance floor and n together. The fix is a larger, shuffled sample and a plain comparison against a model that is known to be unrelated.

An image model outputs 768 numbers per item and a text model 4,096. How does the paper check whether they agree?

Chapter 10

When Emergence Is Not Enough

Test what aligned models still get wrong, and learn which fixes need structure rather than scale

Show a vision and language model a picture of a horse eating grass. Offer two captions: “the horse is eating the grass” and “the grass is eating the horse”. A child picks the first in a blink. Slide 41 asks a plain question with a title and nothing else: Limitations of emergence? The session’s readings answer it from three sides.

Reading 1: models that behave like bags of words

Yuksekgonul, Bianchi, Kalluri, Jurafsky and Zou (ICLR 2023) ask exactly the horse question in “When and why vision-language models behave like bags-of-words, and what to do about it?” (our paper lesson: the bags of words paper). A bag of words is a text treated as a pile of words with the order thrown away. Two sentences with the same words are the same bag.

They built a benchmark called ARO, for Attribution, Relation and Order, with more than 50,000 test cases. Visual Genome Attribution tests whether the model links the right property to the right object (for example, telling “a green cup and a white plate” from the same words with the colours swapped). Visual Genome Relation tests relations such as “eating” and “watching”. The COCO and Flickr30k Order tests shuffle the words of a caption in four ways and ask the model to pick the original from five.

The results are poor where the horse question lives. In the paper, CLIP reaches 62% on attribution, where chance is 50%, and about 60% on relations (the paper’s text says 63%, its appendix 59%). On the order tests, where chance is 20%, CLIP scores 46% on COCO order and 59% on Flickr30k order. Another model, Flava, scores 25% on relations, below chance.

Why does this happen? The authors point at the objective of Chapter 2 and at how we test. Retrieval over big datasets rarely needs order: the right caption for a photo can usually be found from its bag of words alone, because the wrong captions in a batch are about different things. A contrastive loss that is satisfied by shortcuts has no pressure to learn the rest. Standard benchmarks do not notice, because they reward the same shortcut.

The fix is to make the shortcut stop working. Composition-aware hard negatives: for each training caption, build a false caption by swapping words, such as exchanging two nouns, and add it to the batch as a negative. The authors also add “strong alternative” images, one of the 3 nearest neighbours by CLIP similarity for each image, so the batch contains photos that really look alike. Then they fine-tune CLIP ViT-B/32 on COCO with these hard negatives and call it NegCLIP. The scores rise: relations from 63% to 81%, attribution from 62% to 71%, COCO order from 46% to 86%, and Flickr30k order from 59% to 91%. Downstream classification and retrieval stay about the same. The cost is small and the change is one extra column in the loss grid of Chapter 2.

Ask the horse question

Pick a scene. The picture is the true caption’s meaning. The device scores the true caption and its word-swapped twin against it. Switch the encoder and compare.

A toy encoder, not CLIP. Each word gets a seeded random 16 number vector. The bag encoder adds the word vectors, so word order cannot matter. The order-aware encoder multiplies each word vector by a vector for its position before adding. The picture vector is the order-aware encoding of the true caption plus a little noise (or, for the bag encoder, its bag). Scores are cosine similarities.

Reading 2: alignment is not always good

Tjandrasuwita, Ekbote, Ziyin and Liang (2025), in “Understanding the Emergence of Multimodal Representation Alignment” (paper lesson: emergence of alignment), ask when emergence happens and whether it helps. Their summary: alignment emergence and its link to task performance both depend on the data. Two properties matter most: how similar the modalities are, and the balance of redundant against unique information. This is the shared and unique split of Chapters 0 and 5. When the task relies on redundant information, better alignment goes with better performance, as the Platonic hypothesis predicts. When uniqueness dominates, that link breaks down. Pushing alignment higher can even hurt. The paper aims to help a builder decide, for a given dataset and task, whether more alignment is worth chasing. Our reading: measure how redundant and how unique your own task is first.

Reading 3: structure that scale does not give you

Brehmer, de Haan, Behrends and Cohen (TMLR 2025) ask a sibling question in “Does equivariance matter at scale?” (paper lesson: equivariance at scale). An equivariant network has a known symmetry built in: if you rotate the input, the output rotates the same way. The question is whether, with enough data and compute, a plain network learns the symmetry anyway. On a rigid-body benchmark with transformers they find three things. Equivariance improves data efficiency, but a plain model with data augmentation can close the gap given enough training epochs. Compute scaling follows a power law, and the equivariant models do better at every tested budget. And the best split of a budget between model size and training time differs between the two.

Read together, the three papers carry one message for alignment. Scale helps, and agreement can emerge. But a shortcut in the objective (bags of words), a task that lives in the unique part of the data (emergence), or a symmetry the data does not teach (equivariance) will not vanish by itself. Build the structure in, or test for it.

Why can a CLIP style model score well on image-to-text retrieval and still fail to tell “the horse is eating the grass” from “the grass is eating the horse”?

Chapter 11

Field Guide: Alignment on One Page

Review the numbers and terms, run the checklist, read the core code, and find where to go next

You can now retell the lecture from memory. It asked how to find the links between modalities, and gave four answers: align elements explicitly with a contrastive loss, match them globally with assignment or transport, align signals that flow by warping or cutting them, and let alignment emerge on its own, with care about where it fails. The summary slide lists the same four parts.

The one-paragraph summary

Alignment finds which elements of one modality connect to which of another. Links are statistical (association, dependency) or semantic (correspondence, relationship). To compare modalities, two encoders give vectors of the same length, and an alignment function (cosine, kernel, CCA, order) scores them. Contrastive learning pulls true pairs together and pushes false pairs apart; InfoNCE does this over a whole batch, and it lower bounds mutual information, so it captures what the modalities share. Choose views that share just what the task needs (InfoMin), and use fission and self contrast to keep the unique parts. Match whole sets with assignment or soft optimal transport. For continuous signals, warp time with DTW, find change points, or discretise with k-means (HuBERT) or a codebook (VQ-VAE). Independent models also align on their own as they improve, but bag-of-words shortcuts, unique information and missing symmetry mean that scale alone is not enough.

The numbers that matter

WhatNumberWhy it matters
Cosine of (3, 4) and (4, 3)24 / 25 = 0.96, unchanged if one vector is stretchedCosine ignores length; the dot product does not
One InfoNCE row, cosines 0.9, 0.2, 0.1loss 0.0012 at τ = 0.1; 0.37 at τ = 0.5Temperature sharpens or flattens the contest
CLIP training400 million pairs; batch 32,768; τ starts at 0.07A batch gives about 1.07 billion contrasts
Mutual information boundI ≥ log N − L; ln 32,768 = 10.40 nats (15 bits)The batch size caps what a loss can prove
Cat and dog toy, 80% right captions0.193 nats of 0.693 possibleMutual information measures how much one side tells about the other
InfoMin views, ImageNet linear readout73.0% top-1, ResNet-50Views near the sweet spot beat views that share too much
Assignments to check4! = 24; 10! = 3,628,800; 20! ≈ 2.4 × 1018Solve with linear programming or the Hungarian method (about N3)
DTW on A = (1, 2, 3, 3), B = (1, 1, 2, 3)cost 0 with warping; 2 withoutStretching time recovers the match
Speech frames20 ms step, 50 frames a second; “tomato” about 30 frames for 6 phonemesMany frames map to one phoneme
Mutual nearest neighbour chance levelk / n = 10 / 10,000 = 0.001Always quote a score with its chance floor
NegCLIP on AROrelation 63 to 81%; attribution 62 to 71%; COCO order 46 to 86%; Flickr30k order 59 to 91%Hard negatives fix word order cheaply

The core as code

The two pieces below are the heart of the lecture: the symmetric InfoNCE loss that trains an aligned pair of encoders, and the mutual nearest neighbour score that tests whether two spaces agree. They are sketches. img_enc and txt_enc stand for any two encoders that output vectors of the same length.

pythonimport torch, torch.nn.functional as F

def infonce(img, txt, tau=0.07):
    # img: (N, d) image vectors, txt: (N, d) caption vectors; row i of each is a true pair
    img, txt = F.normalize(img, dim=-1), F.normalize(txt, dim=-1)  # length 1: dot = cosine
    logits = img @ txt.T / tau                       # (N, N): every photo against every caption
    target = torch.arange(len(img))                    # the diagonal holds the true pairs
    return (F.cross_entropy(logits, target)             # photo picks its caption
          + F.cross_entropy(logits.T, target)) / 2     # caption picks its photo

def mutual_knn(za, zb, k=10):
    # za: (n, da) from model A, zb: (n, db) from model B; da and db may differ
    def nbrs(z):
        z = F.normalize(z, dim=-1); s = z @ z.T; s.fill_diagonal_(-9)
        return s.topk(k, dim=-1).indices                  # (n, k) neighbour ids inside one space
    a, b = nbrs(za), nbrs(zb)
    shared = (a.unsqueeze(-1) == b.unsqueeze(-2)).any(-1).float().sum(-1)
    return (shared / k).mean()                          # 1.0 = same neighbours; chance is about k / n

Read the first function line by line. Normalising makes every vector length 1, so the matrix product is the cosine grid of Chapter 2. Dividing by τ is the temperature. Cross entropy against the diagonal is −log of the true caption’s share, row by row. Doing it both ways makes the loss symmetric. To add NegCLIP’s fix, append the vectors of word-swapped captions as extra columns of logits, and keep the targets the same.

A checklist before you trust an aligned model

  1. Name the link. Which of the four kinds do you expect: association, dependency, correspondence, relationship?
  2. Check the overlap. How much does the task need that only one modality holds? If it is a lot, add self contrast.
  3. Normalise. Use cosine or unit vectors, so a loud vector cannot win.
  4. Quote the ceiling. The batch size caps the information bound at log N.
  5. Sweep the views. Test several augmentation strengths and pick the peak of the curve.
  6. Test order. Include word-swapped captions in the evaluation, not just retrieval.
  7. Quote chance. Report any neighbour alignment with its chance floor and the sample size.
  8. Check dead codes. For any codebook, plot how often each code is used.

Where this sits

This session is the second core challenge, alignment, of the course’s multimodal map. Slide 4 splits it into three parts: discrete connections, continuous alignment, and contextualized representation.

Keep going

Credit

This lesson follows Lecture 4.2 of MIT MMAI: Multimodal AI (Spring 2026), taught by Paul Liang and the MIT Multisensory Intelligence group. The lecture’s own slides are at the course site; several of them were co-developed with Louis-Philippe Morency for CMU 11-777. Every explanation, number check and device here is our own work, and any mistake is ours.

The takeaway. Alignment is a choice about what to keep. Contrast keeps what the modalities share. Fission keeps what each holds alone. Matching and warping decide which element goes with which. And when alignment appears by itself, test it, because scale will not fix a shortcut, a unique signal or a missing symmetry for you.
Your task needs the texture of a product, which captions rarely mention. Which plan follows the lecture?

Now press Teach and explain, from memory, how a contrastive loss pulls a photo toward its caption, why that loss can only prove log N nats, and what it throws away. Then go back to the circle at the top, set the temperature to Flat, train again, and say why the pairs still meet while the loss stays higher.

MIT MMAI Lecture 4.2: Multimodal Alignment
The course site · Lecture slides · CLIP · The Platonic Representation Hypothesis · Back to Gleams