Two witnesses study a robbery alone. Both saw the car, but only one saw the thief’s face. Their notes can match on the car and never on the face. Two AI models trained apart behave the same way: the data sets how far they can agree, not the size of the models.
Learn how to measure whether two models agree, why the agreement stops short when each model holds facts the other cannot see, and why more agreement does not always mean a better model.
Drag Facts only one model can see from 0 to 8. Watch the two agreement scores fall, because the ceiling falls with it. Then we build, piece by piece, the measures, the synthetic data, the three questions the paper asks and the rule it hands to a builder.
You need a feel for what an embedding is (a list of numbers that stands for a thing) and what a correlation is. We build the rest from zero.
Paper: Megan Tjandrasuwita, Chanakya Ekbote, Liu Ziyin and Paul Pu Liang (MIT and NTT Research), ICML 2025, arXiv 2502.16282. This is a reading for the MIT MMAI course (Multimodal AI, Spring 2026) and belongs to the lecture Alignment. We re-teach every idea in our own words and devices; the paper has the originals.
Illustrative, computed live in your browser. The label needs 8 binary features. Each model sees only its own share, and the score is measured on those features. The setup follows the paper’s synthetic data (Section 3 and Appendix C.1); the paper trains encoders, here the encoder is the ideal one that keeps exactly what the model can see.
Chapter 0
See why two models trained apart might still agree, and why the paper asks whether that agreement helps
Picture two people who have never met. One has only read about animals: thousands of pages, no pictures. The other has only looked at animal photos: thousands of them, no captions. You hand each person ten animals and ask for the same thing: put the animals in a map, so that similar animals sit close together.
Then you lay the two maps side by side. Wolf sits near dog on both. Whale sits near dolphin on both. Neither person ever saw the other’s material. If the maps match, the two people have aligned pictures of the world without ever comparing notes.
A modality is one kind of data: text, images, audio, video, a sensor trace. A representation (also called an embedding) is the list of numbers a model writes down for one input. Think of it as the model’s map position for that input. Two inputs the model finds similar get nearby positions.
Representation alignment asks one question: do two models, usually from two modalities, put the same things close together? The paper opens with this sentence: multimodal learning is about “transforming incomparable modalities into comparable representations”. A text model and an image model write numbers of different lengths, in different spaces. Alignment is what makes them comparable.
The first way is explicit alignment: you train for it. The best known case is contrastive learning, the method behind CLIP (our CLIP lesson builds it). You show the model many matched pairs, such as a photo and its caption. You pull the two embeddings of a matched pair together and push mismatched pairs apart. The training goal itself says “align”.
The second way is implicit alignment: it just happens. You train an image model on images only. You train a language model on text only. Nobody ties them together. Yet recent work found that as the models grow bigger and better, their maps start to agree. This is the surprise the paper starts from.
Huh, Cheung, Wang and Isola (2024) gave the surprise a name: the Platonic Representation Hypothesis. Plato imagined that every object is a shadow of an ideal form. The hypothesis says every model, whatever its data, is learning a shadow of the same underlying reality. Bigger models see more of it. So bigger models should look more alike, and in the end they should converge.
If that is true, a builder gets good news and a puzzle at once. The good news: scale alone might align your modalities for free. The puzzle: explicit alignment methods such as CLIP have worked very well for years. If alignment arrives by itself, why do they help at all? The paper says there are two ways out. Either emergent alignment is not enough, or it does not always lead to better performance. The paper sets out to measure which, and when.
The authors argue the answer to both depends on two properties of the data, not on the models. The first is interactions: how much of the task-relevant information both modalities share (redundant) and how much only one of them holds (unique). A photo and its caption are mostly redundant. A face and a tone of voice, together judging sarcasm, hold much that is unique. The second is heterogeneity: how different the two modalities look to begin with, from two languages (similar) to text and video (very different).
Before we meet the method, try your own guess. Five real datasets from the paper’s benchmark MultiBench sit below. For each, decide: is the information mostly shared between the modalities, or does the task need facts that only one modality has?
Read the card, then choose. After each choice the card shows the correlation the paper measured between alignment and performance for the vision and audio models of that dataset (its Table 1).
The correlations are the paper’s Table 1, vision and audio columns, as the correlation between a model’s alignment and its performance. Which datasets need unique information follows the paper’s own grouping: AVMNIST is the high redundancy benchmark, the other four are affective computing tasks that need unique information in vision, audio and language.
Look at the pattern. On the one dataset where both modalities say the same thing (a picture of a digit and a voice saying the digit), alignment and performance rise together: 0.887 and 0.723. On the tasks about feelings, humour and sarcasm, the numbers are weak, and often negative. The rest of the paper explains why, and then turns the pattern into a tool.
A correlation is a number from −1 to 1 that says how two lists of numbers move together. At 1, when one list goes up the other always goes up. At −1, when one goes up the other always goes down. Near 0, knowing one tells you nothing about the other. The paper uses the Spearman correlation, which first replaces each number by its rank (first, second, third) and then correlates the ranks. It only cares about order, so a few extreme values cannot dominate it.
Why does this matter for a builder? You rarely get to read a model’s inner map. You get a score on your task. If alignment and score rise together, alignment is a handy dial to watch. If they do not, watching alignment can mislead you. The whole paper is a careful test of which case you are in.
The hero at the top of the page is the whole paper in a small box. Its dial is Chapter 2 and Chapter 3. Its two bars are Chapter 1.
Chapter 1
Build the two scores the paper uses, CKA and mutual nearest neighbours, and see what each one ignores
Back to the two map makers. Suppose one draws her map on paper and the other on a tablet turned sideways, at a different zoom. Their coordinates look nothing alike. A point at (3, 1) on one map is at (−1, 2) on the other. Yet the maps may say the same thing: the same animals huddle together.
So a good alignment score must not compare coordinates. It must compare relations: how close is item i to item j in map one, and how close in map two? This is the one idea behind both scores in the paper.
Take n inputs, say n = 1,024 Wikipedia articles, each with a photo and a text. Run the photos through a vision model and the texts through a language model. You get two tables: Z1 ∈ Rn×d1 and Z2 ∈ Rn×d2. Row i of each table is the embedding of input i. The two models may have different widths d1 and d2. That is fine. What matters is that row i in both tables describes the same input.
The score that comes out is one number.
Centered Kernel Alignment (CKA, Kornblith et al. 2019) compares how the rows of Z1 relate to each other with how the rows of Z2 relate to each other. Three steps build it.
Step 1: centre. Subtract each column’s mean, so the cloud of points sits on the origin. Now a shift of the map changes nothing.
Step 2: measure how the two tables vary together. Multiply Z1 transposed by Z2. The result has d1 rows and d2 columns. Entry (a, b) says how strongly feature a of model one rises and falls with feature b of model two across the n inputs. The paper squares and sums every entry of this table. That sum, divided by (n−1)2, is the HSIC (Hilbert-Schmidt Independence Criterion) with a linear kernel. It is large when the two models’ features move together.
Step 3: normalise. A bigger scale gives a bigger HSIC, so divide by how much each model varies with itself. Then the score lies between 0 and 1.
One honest note. The paper’s synthetic plots use the unbiased CKA estimator, which corrects a small-sample bias. We teach the plain linear form of Appendix B, which has the same idea and the same invariances.
A worked case with one feature each and n = 4 inputs. Let the centred columns be Z1 = (−3, −1, 1, 3) and Z2 = (−1, −3, 3, 1).
Z1TZ2 = 3 + 3 + 3 + 3 = 12 Z1TZ1 = 20 Z2TZ2 = 20
CKA = 122 / (20 × 20) = 144 / 400 = 0.36
With one feature, CKA is the squared correlation: 0.6 squared is 0.36. Now double model two: Z2 = (−2, −6, 6, 2) gives a cross term of 24 and a self term of 80. CKA = 576 / (20 × 80) = 0.36 again. The scale cancelled. The same holds if you rotate the whole map. That is the property the paper stresses: CKA ignores rotations and a uniform zoom, so two models with different random starts can still score high.
CKA compares whole covariance structures. For large vision and language models that is costly, and it can be dominated by a few strong directions. The paper, following Huh et al., uses a lighter local score for big models, the mutual k-nearest neighbours (mutual KNN) score.
For each input i, find its k closest inputs in model one’s map, and its k closest in model two’s map. Count how many neighbours appear in both lists. Add this over all inputs and divide by n times k. The answer is the average share of neighbours the two models agree on. (The paper’s Equation 5 also divides by each model’s score against itself. The plain count shows the idea.)
n = 4, k = 1. Model one’s nearest neighbours: (2, 1, 4, 3). Model two’s: (2, 1, 3, 4). Shared: inputs 1 and 2. Score = 2 / (4 × 1) = 0.50
The paper’s recipe for big models: k = 10, 1,024 Wikipedia caption samples, the class token of each vision layer, an average over the tokens of each language layer, l2 normalisation, and values above the 95th percentile clipped. It then compares every layer of one model with every layer of the other and reports the best pair. Chance level is k/n: about 1% here.
Try both scores on a toy. Sixteen points in a plane stand for sixteen inputs. Model two’s map starts as a copy. Rotate it, squash it, or scramble who is paired with whom.
Same colour means the same input. Choose how to damage model two’s map, then drag the slider. Watch which score notices.
Illustrative toy: 16 seeded points in three loose groups, with k = 3 neighbours for the neighbour score. Rotate turns map two by up to 180 degrees. Squash shrinks its vertical axis to as little as 10%. Shuffle swaps the pairing of a growing share of the points. CKA is the linear CKA of the formula above.
Rotation leaves CKA at exactly 1.00 and the neighbour score at 1.00. Squashing changes the shape of the cloud, so CKA drops step by step, to 0.65 at the strongest squash. Shuffling destroys the pairing, and both scores fall to about 0.1 or 0.2. That is the behaviour you want: blind to coordinates, sharp on relations.
Here is a failure mode you may meet. You embed ten thousand product photos with one model and their descriptions with another, and you compute a CKA of 0.31. The symptom is a number that looks low. Is something broken? The measurement is to repeat the score on a copy of the same table with its rows shuffled: if that gives 0.01, the 0.31 is real structure above chance. The fix is not to change the models, but to read the number against the right baseline: chance, and the ceiling the data allows (Chapter 3).
One caveat belongs here. Neither score says why two models agree, and neither says the agreement is useful. They measure only how similar the neighbourhoods are. The rest of the paper asks what shapes that number, and whether it predicts anything.
Chapter 2
Construct the paper’s synthetic world, where you choose exactly how much each modality knows alone
You cannot answer “does uniqueness limit alignment?” on photos and captions. In real data you never know how much of a caption’s meaning is also in its photo. So the authors build a world where they know it exactly, and they turn one knob at a time.
Think of a detective case with two witnesses. Some clues both witnesses saw: that is redundant information, written R. Some clues only witness one saw, U1, and some only witness two saw, U2. These are unique clues. A verdict that needs the unique clues cannot be reached by either witness alone.
In the paper each sample is built from three vectors of random bits (each bit is 0 or 1 with equal chance): a shared vector xr and two private vectors xu1 and xu2 (Appendix C.1). Modality one is the join of xr and xu1. Modality two is the join of xr and xu2. The paper states that each input has 12 features and that 8 of the bits matter to the label. It does not list the split of the 12. For our devices we pick 8 shared bits and 4 private bits per modality, which fits those two facts. The model widths in the paper are also 12.
Not every bit has to matter. A task mask is a vector of 1s and 0s that says which bits feed the label. Multiplying a bit vector by its mask (the Hadamard product, entry by entry) keeps the chosen bits and zeroes the rest. The label is a nonlinear function of the kept bits.
The control is simple. The total number of kept bits, |S|, stays fixed. The paper then defines the uniqueness as U = |S| − R: how many of the kept bits do not come from the shared part. Slide U up and the label leans less on shared bits and more on bits that only one modality has. The label never gets easier or harder to define. Only who can see its ingredients changes. Our devices use |S| = 8, which matches U running from 0 to 8 in the paper’s plots.
A worked sample with U = 2, so R = 6 and U1 = U2 = 1. Say xr = 1 0 1 1 0 1 0 0, xu1 = 0 1 1 0 and xu2 = 1 0 1 0.
kept shared bits: 1 0 1 1 0 1 (first 6), sum 4 kept unique bit of one: 0 kept unique bit of two: 1
total of kept bits = 4 + 0 + 1 = 5, which is odd, so y = 1 (illustrative parity rule)
Model one sees xr and xu1. It knows 6 of the 8 ingredients: the six shared bits and its own one. It is missing the bit that only model two holds. Without that bit it cannot tell y = 1 from y = 0. That is what uniqueness does.
Heterogeneity is how different the two modalities look, whatever the task. In the paper’s synthetic world, modality two is passed through a chain of nonlinear layers, written φ. Each layer is a bijection: a one to one map that loses nothing, so the information content stays the same and only its look changes. The number of layers Dφ is the heterogeneity. With Dφ = 0 the modalities are twins. With Dφ = 9 modality two is a bent, scrambled version of the same facts. Think of the same story in the same language, then translated through nine different codes.
When unique bits exist, the bend applies to the whole second input: X2 = φ([xr, xu2]).
Each modality gets its own encoder, trained on its own, never together. Encoder E1 is a one layer network on modality one. Encoder E2 is an MLP of depth DEnc, from 1 to 10, on modality two. All are MLPs with hidden width 12 and the AdamW optimiser. The learning rate was tuned over {0.1, 0.01, 0.001, 0.0001} and weight decay over {0, 0.1, 0.01, 0.001, 0.0001}. The easy modality trains 50 epochs and the transformed one 300. Alignment is computed on batches of 512 samples.
Everything is repeated with 5 random seeds.
So one experiment is a grid. For each uniqueness U and heterogeneity Dφ, train many encoders of different depths DEnc. For each, record the CKA between E1’s and E2’s features, and the task performance. Then ask how those numbers move together. The next chapters do exactly that.
The paper repeats the idea on real vision and language models. The base is Wikipedia caption data: an image plus its article text, which is heterogeneous but highly redundant. To add unique information, the authors ask GPT-4 to rewrite each caption ten ways, from 10% to 100% unique. The 100% version contains only facts that are not in the image. A sentence BERT model checks that the similarity to the original caption falls steadily as the percentage rises. They keep the original images.
Now turn both dials on a live sample. Choose U, the number of unique kept bits, and Dφ, the number of bending layers. Draw new samples and watch which bits the label can see from each side.
Top row: what model one receives. Middle: what model two receives before the bend. Third row: after the bend. White outlines mark the bits the label uses. The bottom bit is the label.
Illustrative toy. Bits and layout follow the paper (8 shared bits, 4 private bits each, 8 kept bits in total). The label rule is parity, because the paper only says nonlinear. The bend is a fixed random layer plus a leaky nonlinearity, repeated Dφ times; it keeps all information but the paper’s exact layers are not published in the text.
Set U to 8. No kept bit is shared, and the two inputs share only dead bits that the label ignores. Nothing in model one can predict what model two will make of the same sample, and the other way round. That is the stage for Chapter 3.
Chapter 3
Derive why unique facts cap how well two models can agree, then check the cap against the paper’s plots
Two witnesses describe a robbery. Both saw the getaway car. Only one saw the thief’s face, and only the other heard the shout. Compare their notes after each has studied hard for a year. The notes will overlap on the car. They cannot overlap on the face, however much each witness improves, because the other never saw it.
That is the paper’s first answer. Alignment cannot rise above what the two modalities have in common. Bigger and better models can pull the shared part together. They cannot invent the missing part.
The authors vary U from 0 upward. For each U they train many pairs of encoders, with different model sizes and different bend depths Dφ. They plot every pair’s alignment against U (Figure 4). Each dot is one run. The red dots mark the maximum alignment reached at each U, the best any model size could do. The question: does the red line fall as U rises?
It falls in every panel. The paper reports the Spearman correlation between maximum alignment and U (a rank correlation: −1 means the larger one variable, the smaller the other, always). The five panels give −0.950 (Dφ = 1), −0.833 (3), −0.983 (5), −0.983 (7) and −1.000 (9). The same happens on real models (Figure 5): alignment between DINOv2 vision models and large language models, on captions with rising uniqueness, has a maximum that falls with uniqueness, with Spearman −0.950. Appendix D.1 repeats the synthetic test with other metrics (unbiased CKA with linear and RBF kernels, SVCCA, mutual KNN) and with batch sizes 256, 512 and 1024 (Figures 11 to 13), to check that the finding does not rest on one metric.
We take the best case for alignment: a perfect encoder for each modality that keeps exactly the label’s ingredients that the modality can see. Model one keeps R shared bits and U1 private bits. Model two keeps the same R shared bits and U2 private bits. Each bit has variance σ2 and the bits are independent.
Look at the cross table Z1TZ2. A shared bit appears in both models, so its entry is about σ2. A private bit of model one against anything in model two is independent, so its entry is about 0. Squared and summed, the numerator of CKA is Rσ4. The self tables: model one has R + U1 independent bits, so ∥Z1TZ1∥F = σ2√(R + U1). Model two gives σ2√(R + U2).
Work three cases. U = 0: R = 8, so CKAmax = 8 / 8 = 1. U = 4: R = 4, U/2 = 2, so 4 / 6 = 0.667. U = 8: R = 0, so the ceiling is 0. A real trained pair will sit below this ceiling, since training is imperfect. The paper’s point is that the ceiling itself drops, and no amount of capacity lifts it.
Mutual nearest neighbours fall faster. A neighbour list depends on all distances at once, and one missing ingredient moves many neighbours. Try it. The device runs the real scores on freshly drawn data for each U, and compares them with the derivation.
Solid purple: CKA measured live on random samples. Dashed: the ceiling R / (R + U/2) we derived. Teal line: mutual nearest neighbours. Drag U to move the marker. Change the sample size to see the noise.
Computed live from ideal encoders (see the hero note), averaged over 3 random draws per point, with k = 10 for the neighbour score. The paper trains real encoders, reports Spearman correlations of −0.950, −0.833, −0.983, −0.983 and −1.000 between the best alignment and U (Figure 4), and its Figure 6 panel for U = 8 has alignment values between about 0.003 and 0.013 (read from the axis). Our curve shows the same direction.
Read the dashed line against the solid one. They sit together within noise, so the ceiling you derived is the ceiling the data imposes on an ideal pair. At U = 8 both scores sit at the floor.
Suppose you pair a camera embedding with a force sensor embedding and measure a CKA of 0.03. The symptom is a score near zero, and the first thought is that training failed. The measurement is to ask what the label needs. If most of the facts that decide the task are in only one sensor, the ceiling from our derivation is low, and 0.03 is close to it. Even at U = 8 the measured value is not exactly zero: with a finite sample, random bits correlate a little by chance. The paper’s own U = 8 panel in Figure 6 has alignment values of about 0.003 to 0.013, the same small floor. The fix is to stop treating alignment as the goal for this pair. Fuse the two signals instead, and let each keep its own facts.
The reverse symptom matters too. If you expected a low ceiling but measure a high CKA, check for leakage: the same facts may be entering both modalities through a shortcut, such as a timestamp in both files.
The conclusion is a limit, not a refutation. When the two modalities are fully redundant, the paper says its result “supports the Platonic Representation Hypothesis”: capacity can align them. As unique information and heterogeneity grow, the paper calls its result “a generalization of this phenomenon”. A reader building a robot with a camera and a force sensor should expect the two embeddings to stay apart, however large the models. The sensors know different things, and that is exactly why you have both.
Chapter 4
See when a deeper encoder can undo a bent modality, and when unique facts make depth stop helping
Imagine a friend sends you a message. She wrote it, then ran it through five secret codes, one after another. To read it you need to undo five codes. A reader with two decoding steps cannot finish. A reader with five can. A reader with nine has spare steps, and can also read messages with fewer codes.
That is the paper’s picture of heterogeneity and model capacity. Modality two is the message, bent through Dφ nonlinear layers. Its encoder is the reader, with DEnc layers. The paper looks at the gap between them.
Before any encoder trains, ask a plain question. Take the same inputs, and compare them with their bent version. How much do the two feature tables agree? Our device below measures exactly that with CKA, on random bit vectors of width 12 bent by fixed random layers. With zero layers CKA is 1.00. After one layer it is 0.67, after three 0.41, after nine 0.23. Each bend layer scrambles the relations a little more. This is why heterogeneity lowers the achievable alignment: the paper’s Appendix D.1 reports that heterogeneity is negatively correlated with the level of achievable alignment.
Now train encoders. The paper plots alignment against DEnc − Dφ (Figure 6): the encoder’s depth minus the bend depth. A positive gap means the encoder has enough layers, in principle, to undo the bend. A negative gap means it falls short.
The paper measures the Spearman correlation ρ between this gap and alignment, at five uniqueness levels.
| Uniqueness U | Correlation of (DEnc − Dφ) with alignment | What it says |
|---|---|---|
| 0 | 0.844 | Capacity wins: depth undoes the bend |
| 2 | 0.735 | Still strong |
| 4 | 0.432 | Weaker |
| 6 | 0.489 | Weak |
| 8 | −0.049 | No relation at all |
Read the column from top to bottom. With no unique facts, the gap predicts alignment well: 0.844. As unique facts grow, the link fades, and at U = 8 it is gone. The paper’s words: capacity “is effective in handling heterogeneity” when uniqueness is low, but when modalities hold a lot of unique information, “simply increasing model capacity is insufficient”.
A deeper encoder can undo a bend, because a bend keeps all information. It cannot undo missing information. At U = 8 the two inputs have nothing to share, so there is nothing for extra layers to line up. The ceiling from Chapter 3 is already at the floor. Depth cannot lift a ceiling that sits at zero. At U = 0 the ceiling is 1, and the only obstacle is the bend, which depth removes.
One worked reading. Take Dφ = 5. An encoder with DEnc = 2 has a gap of −3. One with DEnc = 5 has a gap of 0. One with DEnc = 9 has a gap of +4. At U = 0 the paper’s trend says these three rank in alignment as the gaps do: −3 lowest, +4 highest. At U = 8 the trend says to expect no order at all.
Chapter 0 introduced the Spearman correlation as a correlation of ranks. Here is one tiny case. Five encoders have gaps (DEnc − Dφ) of −4, −2, 0, 2, 4, which rank 1 to 5. Their alignments are 0.10, 0.18, 0.15, 0.30, 0.41, which rank 1, 3, 2, 4, 5. The ranks move almost together, so ρ is near 0.9. Now shuffle the alignments so the ranks read 3, 5, 1, 4, 2. Nothing lines up, and ρ drops to −0.3, with no clear order left. The paper’s 0.844 is the first kind of picture, and −0.049 is the second.
A failure mode follows. You deepen the second encoder from 4 layers to 10 and the alignment score does not move. The symptom is an expensive change with no gain. The measurement is the unique share of your task: if it is high, the paper predicts a correlation near zero. The fix is to spend the compute on performance instead, where depth keeps paying, and to leave alignment alone.
Not when unique facts are present. Figure 9 fixes the second encoder at its deepest and plots alignment against performance at each bend depth. At low U, shallow bends give higher performance and higher alignment, both together. Past U = 4, the paper says performance is higher at lower synthetic depths, “the alignment scores on these datasets are not necessarily higher”. In the text: models trained on similar modalities do not reliably beat those trained on very different ones. The simple story that more heterogeneity always means less alignment fails once unique facts enter.
The device has two halves. The curve is live: the CKA between a set of inputs and their bent copy as you add layers. The bars are the paper’s Spearman correlations. Choose a bend depth and a uniqueness level.
Top: how alike the inputs stay to their bent copy as layers are added (live). Bottom: the paper’s correlation between (DEnc − Dφ) and alignment for each U. Pick U to highlight its bar.
The curve is illustrative: CKA between 256 random 12-bit inputs and the same inputs through Dφ fixed random leaky layers, computed in your browser. The bars are the paper’s Spearman correlations from Figure 6: 0.844, 0.735, 0.432, 0.489 and −0.049 for U = 0, 2, 4, 6, 8.
Take away a builder’s rule. If your two modalities mostly say the same thing, a bigger encoder is a fair way to improve alignment. If each modality holds facts the other lacks, a bigger encoder will not close the gap, and you should stop expecting it to.
Chapter 5
Compare how alignment and model size each track task performance, and watch alignment lose its grip as uniqueness grows
A coach tracks two numbers for a team: how well the players match their moves, and how many games they win. In a team that wins by passing, the two numbers rise together. In a team of solo stars, the stars can win more while matching less. If you only watch the matching number, you might coach the wrong thing.
The paper asks the same of multimodal models. Many researchers treat alignment as a progress bar: more alignment, better model. Is that safe?
For each synthetic dataset (one U, one bend depth, one seed) the authors train encoders of many depths. Each encoder gives three numbers: its alignment with the other modality’s encoder, its task performance, and its depth (the paper’s stand-in for model size). They then compute Spearman correlations within that dataset, across the depths, and repeat over bend depths and seeds. Figure 7 shows the spread of those correlations for each U as a box: the orange line is the median and the dots are outliers.
Three findings from Figure 7.
The paper’s reading is that alignment is not a universal law. It “is introduced by some special properties of data”. The influence of model size on performance, by contrast, looks universal and fits the usual scaling laws. In summary: “perfect alignment cannot be simultaneously achieved with optimal performance when the data modalities inherently differ in their information content”.
Why would a more capable model align less? The paper does not prove a mechanism, so take this as a hedged picture. A model that is rewarded only for the task keeps whatever helps the task. If the task needs facts only one modality holds, the best model of that modality leans on them, and its map moves away from the other modality’s map. Better performance then means more of the unique part, and less agreement.
Here is the failure mode in practice. The symptom: your team tracks alignment as its progress bar, and it stalls while the accuracy still climbs. The measurement is the correlation of alignment with performance across your trained models, plus the correlation with size. If performance tracks size and alignment does not, the data hold unique facts. The fix is to track performance, and use alignment only as a diagnostic of what the two modalities share.
Does it survive on real models? The paper uses DINOv2 vision models (small, base, large, giant) against language models from three families (BLOOM, OpenLLaMA, LLaMA) from 0.56B to 65B parameters. Performance of the language model is measured as bits per byte, a loss on text: lower is better, so the plots use its negative. For each of three caption sets (10%, 50% and 90% unique), they fit a line of alignment against performance for each vision model size (Figure 8, with the average Spearman correlation).
| Unique share of captions | Slope of alignment on performance | Spearman correlation |
|---|---|---|
| 10% | 0.112 | 0.852 |
| 50% | 0.071 | 0.900 |
| 90% | 0.024 | 0.612 |
The slope is the number to watch. It says how much alignment a language model gains for each unit of performance it gains. At 10% unique it is 0.112. At 90% it is 0.024, a fifth as steep. A better language model still gets better. It just buys much less alignment with each improvement.
A worked comparison. Say a language model improves by 0.20 bits per byte. At 10% unique, alignment to DINOv2 rises by about 0.112 × 0.20 = 0.022. At 50%: 0.071 × 0.20 = 0.014. At 90%: 0.024 × 0.20 = 0.005. Same gain in performance, and the alignment payoff shrinks to about a fifth.
Two more checks keep us honest. First, the paper reports that making the first encoder deeper (Appendix D.2) does not change the synthetic results much. Second, it also repeats the vision-language test with other vision models (Appendix D.5 and D.6). The paper’s summary of the real-model result: “higher-performing models with greater capacity do not necessarily exhibit stronger alignment” when uniqueness is high.
Each line shows alignment to DINOv2 gained as the language model improves, using the paper’s fitted slopes. Choose a caption set, then drag how much the model improves.
The slopes (0.112, 0.071, 0.024) and the average Spearman correlations (0.852, 0.900, 0.612) are from the paper’s Figure 8. The 0.30 bits per byte span is our illustrative range, and the lines start at zero for clarity; the paper fits a separate line for each vision model size.
Put the two halves together. In the real and the synthetic tests, scale reliably improves performance. Scale improves alignment only while the modalities share what the task needs. So a team that tracks only alignment can mistake a stall in agreement for a stall in quality.
Chapter 6
Read the paper’s MultiBench table cell by cell and learn what a weak or negative correlation tells you
Synthetic data can show a mechanism. A builder works with real data, where nobody sets U. So the third question asks: can we measure the link between alignment and performance for a given dataset, and does it match what the synthetic world predicts?
The paper uses MultiBench (Liang et al. 2021), a collection of real multimodal datasets. Five are used. MOSEI asks for the emotion of a speaker in a video. MOSI asks for the sentiment of a spoken movie review. URFUNNY asks whether a talk contains humour. MUStARD asks whether a TV clip is sarcastic. These four use vision, audio and language features. AVMNIST asks for the digit, from an image of the digit and a recording of it being spoken.
For each modality the authors train transformers. Each transformer takes the pre-extracted features of one modality as a sequence of vectors. The embedding width equals the input width, with one attention head. For classification, a learnable [cls] token is added to the sequence. Its final embedding is what alignment is computed on. For other tasks they average over the sequence. Depths run from 1 to 10 layers, for three random seeds, which gives 30 trained models per modality, so 30 alignment and performance points.
One subtle choice keeps the test fair. Take modality one. Pick its single best model by validation score across seeds. Then compute the alignment of every modality two model to that one reference. Finally correlate those 30 alignment scores with the 30 performance scores of the modality two models. The learned features go in. One correlation per modality per pair comes out.
There are three pairs: vision and audio, vision and text, and audio and text. Each pair gives two correlations, one for each modality’s performance. The paper’s Table 1 has 26 numbers.
The affective datasets are the ones the paper says need unique information from vision, audio and language. Look at them first. Of their 24 correlations, 13 are negative, and none is above 0.438. MOSEI is negative in all six cells, from −0.093 to −0.641 for the text model paired with audio. For MOSEI’s audio and text pair, the more a text model aligns with audio, the worse its performance. AVMNIST, the redundant task, is the opposite: 0.887 for vision and 0.723 for audio.
The relation also changes within a dataset. On MUStARD the paper notes that alignment correlates more with vision performance, while audio and text performance look less tied to it. On MOSI the audio model aligned to vision correlates at 0.313, while the vision model correlates at −0.014. The same two modalities can disagree about whether alignment helps. Different tasks, and different sides of the same task, use different amounts of unique information.
The paper’s own summary: on the affective tasks “the correlation is weak and often negative, suggesting that enforcing alignment between modalities may not be desirable”. On AVMNIST “the alignment of vision and audio modalities … is highly correlated with performance”.
Each row is a dataset. Two bars per row: the correlation for each modality’s performance. Teal bars help, warm bars point the wrong way. Choose a pair, then tap a dataset.
All 26 numbers are the paper’s Table 1. AVMNIST has only a vision and audio pair. Bars use one shared scale from −1 to 1. Correlation here comes from 30 trained models per modality, so one number can shift with another seed set; compare the pattern, not the third decimal.
Here is a tiny version of the computation, with five models and made-up numbers. The five alignment scores are 0.20, 0.25, 0.22, 0.30, 0.28. Their accuracies are 61, 60, 63, 59, 58 percent. As alignment rises, accuracy tends to fall. The ranks of the two lists run nearly opposite, so the Spearman correlation is about −0.8. That is what a negative cell in Table 1 means. It is a statement about a pattern across the models you trained.
The failure mode for a team: the symptom is that your aligned multimodal model does worse on a sarcasm task than a plain fusion model. The measurement is this table’s method on your own models and data. If the correlation is flat or negative, the paper’s advice is that enforcing alignment may not be desirable. The fix is to keep the modalities less tied, or to use methods built to keep unique and synergistic information, such as the factorized contrastive learning the paper cites (Liang et al. 2023).
A correlation across 30 models is not a cause. A positive value does not prove that pushing alignment up will help; Chapter 7 tests that step directly. All models here are small transformers on pre-extracted features, so large pretrained encoders may differ. And the amount of unique information in each dataset comes from earlier human annotation work (Liang et al. 2023), not from a new measurement. The paper says it cannot vary the information content of real datasets, so the match to the synthetic story is supportive evidence, not proof.
Chapter 7
Turn the alignment and performance slope into a dial for the weight of a CLIP loss on MM-IMDb
You have a million movie posters with plot summaries, and only a thousand of them carry a genre label. Labels cost money. Pairs are free, since each poster already comes with its plot text. You want to use the free pairs to help a genre classifier. One good tool: add a contrastive loss that pulls each poster close to its own plot.
But how hard should it pull? Too little wastes the free pairs. Too much forces the models to agree on things they should keep apart. This chapter shows the paper’s answer: measure how alignment relates to performance before you train, and let the slope set the weight.
The CLIP loss (Radford et al. 2021) works on a batch of B matched pairs. Embed the B images into a table of shape [B, d] and the B texts into [B, d], then normalise each row. Multiply the two tables to get a [B, B] grid of similarities. The diagonal holds the true pairs. The loss rewards a high diagonal and low off-diagonal values, in both directions: each image must pick its text, and each text its image. This is the contrastive learning of Chapter 0, and our CLIP lesson derives it in full.
The paper then mixes it with the task loss:
A worked example makes the loss concrete. Take three posters and their three plots. Row one of the similarity grid, scaled by a temperature, is (2.0, 0.5, 0.1): poster one against plots one, two and three. A softmax turns the row into chances: e2.0 = 7.39, e0.5 = 1.65 and e0.1 = 1.11, which sum to 10.15. The true plot gets 7.39 / 10.15 = 0.73. The loss for this row is −ln(0.73) = 0.32. If the grid were flat, (1, 1, 1), each chance would be 0.33 and the loss would be 1.10. The loss falls as the true pair stands out. Average over rows and columns, and you have LCLIP.
The data is MM-IMDb: movie posters and plot summaries, with 23 genres. Each genre is a yes or no task, so 23 binary tasks. The authors keep 1,024 labelled examples each for train, validation and test, out of 25,000, to mimic scarce labels. They fine-tune CLIP’s vision and language encoders for 30 epochs, with a learning rate of 10−4, a cosine schedule down to 10−6 and a 10 epoch warm-up.
Now the clever part. For ten genres they first measure, with unimodal models that were never aligned, the slope of the line that fits alignment against performance (the same fit as Chapter 5). Performance is the accuracy of a linear classifier on the language embeddings. The slope tells how much performance tends to rise when alignment rises, for that genre.
The hypothesis: where the slope is high, a bigger w should help. Where the slope is low or negative, a bigger w should not.
| Genre (class id) | Alignment and performance slope | Result as w grows (paper, Figure 10) |
|---|---|---|
| 13 | −0.094 | Worse or no gain |
| 19, 10 | 0.049, 0.050 | Worse or no gain |
| 6, 2, 15 | 0.067, 0.068, 0.080 | Worse or no gain |
| 1, 16, 0, 17 | 0.090, 0.114, 0.121, 0.136 | Higher w generally improves |
The split is clean. The six genres with slopes up to 0.080 do not gain from more CLIP weight. The four with slopes of 0.090 and above do. The paper also shows that training with only Lsup overfits the 1,024 labels, which is why the free pairs are worth using at all.
The decision rule that follows is cheap. Before you spend GPU hours on a sweep of w, fit the slope with unimodal models you already have. If the slope is steep, try a large w. If it is flat or negative, keep w small. In a worked case: genre 17 has a slope of 0.136, so give it a large weight such as 10. Genre 13 has a slope of −0.094, so keep w near 0.1, or at 0.
Top: the paper’s ten MM-IMDb genres, sorted by slope. Choose a genre, then drag the weight w. The bottom bar shows how the total loss splits between the task and alignment.
The slopes and the w grid are the paper’s (Figure 10). Which genres improve or do not is the paper’s finding. The loss share w / (1 + w) is plain arithmetic about the weights, not a measured gain. The paper’s per-genre curves are noisy; we show the verdict, not the curves.
Two things this does not show. The slope comes from a small set of unimodal models, and the improvement is a trend across a noisy grid of w, not a guarantee for each single run. The paper treats the result as a demonstration that quantifying the relation helps decide how much to align, with unimodal models that are not explicitly aligned.
Chapter 8
Review the numbers, run the measures in code, and learn where the claim stops
You can now reproduce the paper from memory. It starts from a surprise: independent models can drift toward the same map. It asks two questions. When does that happen? Is it a good sign? It builds a synthetic world with two dials, uniqueness and heterogeneity, and answers with measurements.
Alignment is measured by CKA (relations between inputs, blind to rotation and zoom) or by mutual nearest neighbours (shared local neighbourhoods). The most alignment two modalities can reach is capped by the data: as unique task information grows, or as the modalities grow more heterogeneous, the cap falls. Bigger models improve performance at every uniqueness level. They improve alignment only while the modalities share what the task needs. So alignment predicts performance well on redundant data (AVMNIST) and weakly, sometimes negatively, on data with unique information (the affective tasks). The slope of that relation, measured with unimodal models, can set how much explicit alignment, such as a CLIP loss, to add.
| What | Number | Why it matters |
|---|---|---|
| Max alignment against U, synthetic (Fig 4) | Spearman −0.950, −0.833, −0.983, −0.983, −1.000 | The ceiling falls as unique facts grow |
| DINOv2 and LLMs against uniqueness (Fig 5) | Spearman −0.950 | Same fall on real models |
| Relative depth against alignment (Fig 6) | 0.844, 0.735, 0.432, 0.489, −0.049 for U = 0, 2, 4, 6, 8 | Depth helps only when data are redundant |
| Alignment against performance (Fig 7) | Median near zero and negative cases for U above 3 | Alignment is not a universal signal |
| Slope of alignment on LLM performance (Fig 8) | 0.112, 0.071, 0.024 at 10%, 50%, 90% unique | Better models buy less alignment |
| MultiBench correlations (Table 1) | AVMNIST 0.887 and 0.723; the four affective sets from −0.641 to 0.438 | The link is a property of the dataset |
| MM-IMDb slopes (Fig 10) | −0.094 to 0.080 (no gain from more w); 0.090 to 0.136 (gain) | The slope predicts the right CLIP weight |
| Mutual KNN settings (Appendix B) | k = 10, 1,024 samples, 95th percentile clip | To reproduce the real model scores |
| Term | In plain words |
|---|---|
| Representation alignment | Two models put the same inputs close to the same other inputs |
| Redundant, R | Task information both modalities hold |
| Unique, U | Task information only one modality holds |
| Heterogeneity, Dφ | How bent one modality looks relative to the other, with no loss of information |
| Explicit alignment | A loss that pulls matched pairs together, such as CLIP |
| Implicit alignment | Agreement that appears without such a loss |
A sketch of the three pieces you need: the alignment measures, the synthetic world, and the slope that guides w. It follows the paper’s Appendix B and C.1; the labelling rule here is parity, our choice.
pythonimport numpy as np def cka(Z1, Z2): # Z1: [n, d1], Z2: [n, d2], row i = input i Z1, Z2 = Z1 - Z1.mean(0), Z2 - Z2.mean(0) # centre the columns hs = lambda A, B: np.linalg.norm(A.T @ B, "fro") ** 2 return hs(Z1, Z2) / np.sqrt(hs(Z1, Z1) * hs(Z2, Z2)) # Eq. 2, scale cancels def mutual_knn(Z1, Z2, k=10): def nn(Z): Z = Z / np.linalg.norm(Z, axis=1, keepdims=True) D = ((Z[:, None] - Z[None]) ** 2).sum(-1); np.fill_diagonal(D, np.inf) return np.argsort(D, axis=1)[:, :k] # [n, k] neighbour ids a, b = nn(Z1), nn(Z2) hit = sum(len(set(a[i]) & set(b[i])) for i in range(len(a))) return hit / (len(a) * k) # share of neighbours both models agree on def make_world(n, U, S=8, nR=8, nU=4, seed=0): rng = np.random.default_rng(seed); R = S - U xr, u1, u2 = (rng.integers(0, 2, (n, m)) for m in (nR, nU, nU)) kept = np.hstack([xr[:, :R], u1[:, :-(-U // 2)], u2[:, :U // 2]]) y = kept.sum(1) % 2 # parity: our nonlinear rule return np.hstack([xr, u1]), np.hstack([xr, u2]), y # x1 [n,12], x2 [n,12], y [n] def slope(align, perf): # one value per trained model return np.polyfit(perf, align, 1)[0] # steep: raise w. flat or negative: keep w small
Run make_world for U from 0 to 8, train a small encoder per modality, and plot cka against U. You will see the red line of Figure 4 fall. The authors’ own code is at github.com/MeganTj/multimodal_alignment.
This paper is a reading for the lecture Alignment of the MIT MMAI course. The lecture teaches how to align modalities. This paper asks when you should. Its limits are the ones to carry away.
Most results use small synthetic MLPs and small transformers on pre-extracted features. The label rule and bend layers in our devices are our choices where the paper gives none. Alignment and performance correlations come from tens of trained models. The unique content of real datasets was estimated by earlier annotation work. And the MM-IMDb test covers ten genres, in a scarce label setting. A conclusion for a frontier model needs its own measurement.
Now press Present or Teach and explain the paper out loud, from memory: what CKA measures, why unique facts lower the ceiling, why depth stops helping, what Table 1 shows, and how the slope sets the CLIP weight. Then go back to the dial and set U to 4 to see the ceiling for yourself.