Six modalities — image, text, audio, depth, thermal, IMU — share a single embedding space, trained only on image-paired data, yet they align across modalities that were never paired. Audio can retrieve text; thermal can retrieve audio. The alignment emerges.
You're building a search engine for a robot's sensor logs. The robot recorded six kinds of data simultaneously: an RGB image, a depth map, a thermal frame, an IMU motion trace, an audio clip, and a human's text caption. You want to type "a door slamming" and pull back the matching audio clip, the matching thermal frame, and the matching depth map — even though nobody ever wrote a caption describing the thermal frame, and nobody ever paired the audio with the depth map.
The textbook answer is contrastive learning. CLIP (Radford et al., 2021) taught a single shared space for image and text by training on 400 million (image, text) pairs scraped from the web. Type "a dog," get back dog photos. Beautiful. But CLIP binds exactly two modalities, and it works only because the internet is overflowing with image-text pairs.
Now try to extend that recipe to six modalities. The naive plan is to collect paired data for every modality combination. How many combinations? For n modalities you need a pair for each of the n(n−1)/2 unordered combinations.
Fifteen datasets. And not just any datasets — you would need a large, web-scale collection of, say, (thermal, IMU) pairs. That dataset does not exist. Who labels thermal frames with the matching accelerometer trace? Almost nobody. The combinatorics of pairwise collection is the wall the field hit. This is the pairing problem: most modality pairs have little or no naturally co-occurring paired data.
Drag the slider to add modalities. The left bar (red) is the pairwise plan: n(n−1)/2 datasets, all needed. The right bar (teal) is ImageBind's plan: only n−1 image-paired datasets. Watch the red bar explode quadratically while the teal bar grows by one.
ImageBind's bet is that you don't need the 15 datasets. You need only the datasets that already exist in abundance: each non-image modality paired with images. Images co-occur with text (web), with audio (video frames + their soundtrack), with depth (RGB-D sensors), with thermal (co-registered cameras), with IMU (egocentric video + the device's motion). Bind everything to the image, and — this is the headline result — the modalities end up bound to each other too, without ever being trained on each other's pairs.
| Approach | Datasets needed (n=6) | Trains modality→modality? | Feasible at scale? |
|---|---|---|---|
| Pairwise contrastive | 15 (all combos) | Yes, directly | No — most pairs don't exist |
| One joint model on all-at-once tuples | 1 (6-aligned tuples) | Yes | No — 6-aligned data is vanishingly rare |
| ImageBind (image-anchored) | 5 (each modality × image) | No — only modality→image | Yes — these datasets are plentiful |
The rest of this Veanor builds the mechanism from zero: which encoder produces each modality's vector (Ch. 2), the exact loss that does the binding (Ch. 3, the InfoNCE objective derived line by line), why the alignment emerges for unpaired pairs (Ch. 4), what the temperature knob does (Ch. 5), and the headline numbers and ablations (Ch. 6–7). Two Code Labs let you build the image-anchored loss and the emergent-retrieval step yourself.
Let's start with the single design decision the whole paper rests on: making the image the hub.
Think of the embedding space as a city, and each modality as a neighborhood that wants to live close to its semantic neighbors. The pairwise plan would build a road between every pair of neighborhoods — expensive, and many of those roads have nowhere to start because the land (paired data) doesn't exist.
ImageBind's plan is a hub-and-spoke layout. There is one central plaza — the image — and every other neighborhood builds a single road to the plaza. Audio → image. Text → image. Depth → image. Thermal → image. IMU → image. Five spokes, no perimeter roads. And yet, because everyone routes through the same plaza, any two neighborhoods become reachable from each other.
Why is the image the natural hub? Because images are the modality that co-occurs with everything else through natural data-collection processes:
Crucially, the image (and its naturally paired text) act as a kind of universal currency. The paper notes that web images already have a huge amount of text associated with them, so binding a new modality to images indirectly binds it to text as well. You get the (modality, text) alignment for free, riding on the image as an intermediary.
Left: the full mesh — every modality paired with every other (the 15 edges you'd need). Right: ImageBind's hub — five spokes to the image. Click a spoke node on the right to highlight a trained edge; the dashed orange edge that lights up between two non-image nodes is the emergent path that was never trained.
Let's make the savings concrete for the paper's six modalities (image, text, audio, depth, thermal, IMU).
Full mesh: choose 2 of 6 modalities = 6·5/2 = 15 paired datasets. Each must be web-scale. The (depth, thermal), (thermal, IMU), (audio, depth) datasets effectively do not exist.
Image hub: the image pairs with each of the other 5 = 5 paired datasets. Every one of these already exists at scale. We trade 15 nonexistent datasets for 5 plentiful ones, and (Ch. 4) we recover most of the cross-modal capability anyway.
But for the geometry to work, every modality must end up as a vector in the same space — same dimensionality, comparable scale. That is the job of the encoders, which we turn to next.
Each modality has its own encoder that turns raw input into a single d-dimensional vector. The trick that makes six wildly different signals comparable is uniform: every encoder is a Transformer (or close to one), and every modality's input is first converted into a sequence of tokens that a Transformer can ingest.
Let's trace the data flow, with shapes, for each modality. We write the final embedding dimension as d. After the encoder produces a vector, ImageBind applies a small modality-specific projection head (a linear layer) to map every modality into the same d-dimensional space, then L2-normalizes so all embeddings live on the unit sphere.
| Modality | Raw input | Tokenization | Encoder |
|---|---|---|---|
| Image / Video | RGB pixels [3, 224, 224] | 16×16 patches → 196 tokens | ViT (the image is the hub; encoder initialized from a pretrained vision-language model) |
| Text | Token ids | Byte-pair tokens | Transformer text encoder (CLIP-style) |
| Audio | 2-second clip | Mel spectrogram → 2D patches | ViT over the spectrogram "image" |
| Depth | 1-channel depth map | Disparity, treated as a 1-channel image → patches | ViT |
| Thermal | 1-channel thermal map | Treated as a 1-channel image → patches | ViT |
| IMU | Accel + gyro time series [6, T] | 1D convolution into temporal tokens | Transformer over the IMU token sequence |
Take a batch of B image-audio pairs from an audio-video dataset. Here is the journey, with shapes:
Two engineering decisions are load-bearing here, and the paper is deliberate about both:
1. Freeze the image and text encoders; train the rest. The image encoder is the anchor — the fixed point the whole space is defined against. If the image embeddings drifted while you were also moving audio toward them, you'd be aiming at a moving target and could collapse the space. By keeping the image (and text) encoder frozen and only learning the new modality's encoder + projection, every spoke is fitted against a stable hub. This is also why ImageBind can reuse a strong pretrained vision-language model unchanged.
2. L2-normalize, so similarity is cosine. After projecting to d dimensions, each embedding is divided by its norm. On the unit sphere, the dot product q·k equals the cosine of the angle between them, ranging in [−1, 1]. This makes scores comparable across modalities (no modality can "shout louder" with a bigger norm) and makes the temperature (Ch. 5) the sole controller of sharpness.
Click a modality to watch its data flow: raw signal → tokens → encoder → projection to shared d → L2-normalize onto the sphere. Note how all paths converge to the same final vector shape [d], living on the same unit sphere.
Suppose before normalization audio vector a = [3, 4] and image vector i = [6, 8]. They point in the same direction (i = 2a), so they're perfectly aligned, but the raw dot product is a·i = 3·6 + 4·8 = 18 + 32 = 50 — a big number driven by magnitude, not direction.
Normalize: ||a|| = √(9+16) = 5, so â = [0.6, 0.8]. ||i|| = √(36+64) = 10, so î = [0.6, 0.8]. Now â·î = 0.36 + 0.64 = 1.0 — the cosine, capped at 1, telling us they are perfectly aligned regardless of how long the raw vectors were. Every modality now competes on direction alone.
python import torch import torch.nn.functional as F # Per-modality encoder + projection to a shared dim d class ModalityTower(torch.nn.Module): def __init__(self, encoder, enc_dim, d, frozen=False): super().__init__() self.encoder = encoder self.proj = torch.nn.Linear(enc_dim, d, bias=False) if frozen: # image / text towers are frozen for p in self.encoder.parameters(): p.requires_grad = False def forward(self, x): feats = self.encoder(x) # [B, enc_dim] z = self.proj(feats) # [B, d] -> shared space z = F.normalize(z, dim=-1) # [B, d] -> unit sphere return z # cosine(z_i, z_j) = z_i . z_j
Everything so far has been setup. The actual binding is done by a single loss applied to each (modality, image) pair: the InfoNCE contrastive loss (van den Oord et al., 2018), exactly as CLIP uses it. We'll derive it from scratch, then read it as the engine of the emergence in Ch. 4.
Setup: a batch of B aligned pairs. For a modality M paired with image I, the encoders produce normalized vectors qi (the M-embedding of example i) and kj (the image-embedding of example j). Because each row is L2-normalized, the similarity is just the dot product:
For modality example i, its positive is the matching image, ki (same index = a true pair). All other images kj (j ≠ i) in the batch are negatives — they came with different examples, so they should not match. The task: given qi, pick its true image out of the B candidates. That's a B-way classification, and the natural loss for "pick the right one out of B" is the cross-entropy of a softmax.
We scale the similarities by a learnable temperature τ > 0 and softmax over the B image candidates:
P(j | i) is the model's belief that image j is the true partner of modality example i. Dividing by a small τ sharpens the distribution (Ch. 5); τ controls how harshly mismatches are penalized.
The correct answer for query i is j = i. Cross-entropy with a one-hot target keeps only the −log of the probability assigned to the true class:
Expand the log of the quotient into a difference — this is the form that makes the dynamics readable:
Read it as a tug-of-war. The first term, −sii/τ, falls when the positive similarity sii rises — it pulls the true pair together. The second term, the log-sum-exp over all candidates, falls when the off-diagonal similarities sij (j≠i) drop — it pushes the negatives apart (it's a soft maximum dominated by the largest, i.e. the hardest, negative). Minimizing the loss does both at once.
The loss above asks "given the modality, find the image." By symmetry we also want "given the image, find the modality" — softmax down the columns instead of across the rows. The full InfoNCE loss for the (M, I) pair is the average of both directions, summed over the batch:
And ImageBind simply sums this paired loss over every modality–image dataset it trains on (audio-image, text-image, depth-image, thermal-image, IMU-image). There is no term that pairs two non-image modalities. That absence is the entire story of Chapter 4.
Take B = 2: one (audio, image) example for a "dog bark" and one for a "door slam." Suppose after normalization the cosine similarity matrix S (rows = audio queries, cols = image keys) is:
Scale by 1/τ = 2: S/τ = [[1.8, 0.2], [0.4, 1.6]]. Row 0 (the dog-bark audio): exp(1.8) = 6.05, exp(0.2) = 1.22, sum = 7.27. P(0|0) = 6.05/7.27 = 0.832. Loss contribution −log(0.832) = 0.184. Row 1 (door-slam audio): exp(0.4)=1.49, exp(1.6)=4.95, sum=6.44, P(1|1)=4.95/6.44=0.769, −log(0.769)=0.263. The audio→image direction averages (0.184 + 0.263)/2 = 0.224. The image→audio direction (softmax down columns) is computed the same way and averaged in. Lower temperature would push P(0|0) closer to 1 and shrink the loss further — but also make it brutal whenever the model is wrong.
python import torch import torch.nn.functional as F def infonce(q, k, tau=0.07): """q, k: [B, d], already L2-normalized. Index i in q pairs with index i in k.""" B = q.size(0) logits = (q @ k.T) / tau # [B, B] scaled cosine sims targets = torch.arange(B, device=q.device) # true partner is the diagonal loss_q2k = F.cross_entropy(logits, targets) # modality -> image loss_k2q = F.cross_entropy(logits.T, targets) # image -> modality return 0.5 * (loss_q2k + loss_k2q) # ImageBind total loss = sum of the paired loss over every (modality, image) dataset total = (infonce(q_text, k_img) + infonce(q_audio, k_img) + infonce(q_depth, k_img) + infonce(q_thermal, k_img) + infonce(q_imu, k_img)) # note: there is NO infonce(q_audio, q_text) term. The cross-modal # alignment is never trained -- it emerges (Chapter 4).
Here is the payoff. We trained (audio ↔ image) and (text ↔ image), and never once showed the model an (audio, text) pair. Yet at test time, ImageBind can take an audio clip and retrieve the matching text — audio→text retrieval that was never in the loss. The paper calls this emergent alignment. Let's see exactly why it happens.
Consider one underlying concept — say "dog barking." It appears as an audio clip a, an image i, and a text caption t. After training:
The audio loss made a sit close to its image: cos(a, i) is large, so the angle α between them is small.
The text loss made t sit close to the same image: cos(t, i) is large, so the angle β between t and i is small.
Both a and t are pinned near the same anchor i. How far apart can a and t be? On the unit sphere, angles obey a triangle inequality:
If α and β are each small (the contrastive loss made them small), then their sum is small, so a and t are forced to be close — even though no loss term ever compared them directly. The image is a hinge; binding both arms to the hinge keeps the arms near each other.
Say training drove the audio–image cosine to cos(α) = 0.95 (α ≈ 18°) and the text–image cosine to cos(β) = 0.95 (β ≈ 18°). Worst case, a and t sit on opposite sides of i, so angle(a, t) ≤ 18° + 18° = 36°, giving cos(a, t) ≥ cos(36°) ≈ 0.81. Even in the worst arrangement, the unpaired audio and text are guaranteed cosine ≥ 0.81 — strongly aligned. In practice they often share the same side of the anchor, so the realized similarity is higher still.
The image anchor (blue) is fixed. Drag the two sliders to set how tightly the audio (warm) and text (green) are bound to the image. The dashed orange arc is the untrained audio–text angle. Watch: as you tighten both spokes, the emergent audio–text similarity (shown live) shoots up — without anyone training that edge.
The emergence is bounded by how well each spoke is trained. If the audio–image alignment is weak (large α) or the text–image alignment is weak (large β), the triangle bound is loose and emergent audio–text quality degrades. Emergent performance is always somewhat below what you'd get from directly training the pair, because the bound is an inequality, not an equality. ImageBind's empirical finding is that even this indirect alignment is strong enough to set state-of-the-art emergent zero-shot numbers on tasks like audio classification — without any audio-labeled supervision.
The temperature τ in the softmax is small (ImageBind uses values around 0.05, with per-modality tuning), and it has an outsized effect on what the space learns. Let's understand it as a single dial that controls how sharply the contrastive loss separates positives from negatives.
Recall the per-query distribution P(j|i) = softmax(sij/τ). The similarities sij live in [−1, 1]. Dividing by τ rescales them to [−1/τ, 1/τ] before the softmax. With τ = 0.05 that's [−20, 20] — a huge dynamic range, so even small similarity differences become enormous logit gaps, and the softmax becomes nearly one-hot.
Suppose a query's three candidate similarities are [0.9, 0.6, 0.1] (positive first).
τ = 0.5 (1/τ = 2): logits [1.8, 1.2, 0.2]. exp: [6.05, 3.32, 1.22], sum 10.59. Softmax: [0.571, 0.314, 0.115]. The positive wins, but the second candidate still gets 31% — a gentle gradient.
τ = 0.05 (1/τ = 20): logits [18, 12, 2]. exp: [6.6e7, 1.6e5, 7.4], normalized: [0.9976, 0.0024, ~0]. The positive takes essentially all the mass. The loss is tiny when right — but if the positive had been the second-best score, the loss would be brutal. Small τ demands the model be confidently correct.
Five candidate similarities feed a softmax. Drag τ from 1.0 (flat, hedging) down to 0.02 (peaky, one-hot). The bars are the attention/probability over candidates; the number below is the loss if the leftmost (warm) bar is the true positive. Watch how low τ sharpens the distribution and shrinks the loss — when the model is right.
The right sharpness differs by modality. Text–image similarities are well-calibrated (CLIP tuned them), but a noisy modality like IMU or thermal may need a gentler τ so the loss doesn't catastrophically punish honest mistakes early in training. Rather than guess, ImageBind treats τ (or its log) as a parameter the optimizer tunes, and the paper ablates fixed vs learnable temperature, finding the choice matters for the harder modalities. The result is a per-modality dial, automatically set, that balances "be decisive" against "be forgiving."
The claims would be empty without numbers. ImageBind's evaluation is built to test one thing above all: the emergent zero-shot capability on modality pairs that were never trained together. We'll describe the results as the paper frames them.
The flagship result is emergent zero-shot recognition. Take audio: ImageBind never trained on (audio, text) pairs and never used any audio class labels. To classify a sound, it embeds the audio clip and embeds each candidate text label ("dog barking," "siren," ...), then picks the label with the highest cosine. Because audio was bound to images and text was bound to images, audio and text share the space — so this works zero-shot. ImageBind reports strong emergent zero-shot accuracy on audio benchmarks (e.g. ESC, Audioset), competitive with or exceeding prior methods that used direct audio–text supervision.
The same emergent recipe extends to depth, thermal, and IMU: classify a depth map or thermal frame or motion trace purely by comparing it to text labels through the shared space, with no labeled training data for that modality. This is the headline — capabilities appear for modality pairs with zero paired training data.
| Task | How it works | Supervision used |
|---|---|---|
| Audio zero-shot classification | cos(audio, text-label) via shared space | None for (audio,text) — emergent |
| Depth zero-shot classification | cos(depth, text-label) | None for (depth,text) — emergent |
| Thermal zero-shot classification | cos(thermal, text-label) | None for (thermal,text) — emergent |
| Cross-modal retrieval | audio→image, audio→text, text→audio | Only modality–image at train time |
Because all modalities share one space, you can retrieve across any pair: query with audio, retrieve images; query with text, retrieve audio; query with a depth map, retrieve text. The paper also shows embedding arithmetic — adding an image embedding of a fruit to an audio embedding of birdsong retrieves images that combine both concepts — demonstrating the space is genuinely composed of semantically meaningful, additive directions, much like word2vec analogies but across modalities.
A practical headline: because ImageBind binds new modalities to the image space, you can swap in the ImageBind audio encoder to give an existing image-and-text system a new sense. The paper shows that a text-to-image generator or a detector built for images can be made to respond to audio prompts by routing the audio through ImageBind's shared embedding — no retraining of the downstream model. The shared space is a universal adapter.
A query (left) is embedded once. Candidate labels/items of ANY modality (right) are embedded and ranked by cosine. Click a query modality; the top match lights up. None of the query–candidate pairs shown here were trained directly — they all route through the image-anchored shared space.
An audio embedding a = [0.6, 0.8] (normalized). Three text labels: "dog" t₁ = [0.7, 0.7], "siren" t₂ = [−0.8, 0.6], "rain" t₃ = [0.1, 0.99]. Cosines: a·t₁ = 0.42+0.56 = 0.98; a·t₂ = −0.48+0.48 = 0.00; a·t₃ = 0.06+0.79 = 0.85. ImageBind predicts "dog" (0.98), with "rain" a plausible runner-up and "siren" rejected — all without ever training an (audio, text) pair.
A paper's ablations are where you learn which design choices are load-bearing and which are window dressing. ImageBind's ablations probe the encoder scale, the training data, the projection, and the temperature. We summarize the qualitative findings.
The single biggest lever is the capacity of the image encoder. A stronger, larger pretrained vision encoder (the hub) lifts emergent performance across every downstream modality — audio, depth, thermal, IMU all improve when the image tower gets better. This is the hub-and-spoke logic made empirical: a better plaza makes every neighborhood better connected. Scaling the new-modality encoders helps far less than scaling the anchor.
| Ablation | Finding | Why it makes sense |
|---|---|---|
| Image encoder capacity | Bigger anchor → better on ALL modalities | Every spoke is fitted against the hub; a stronger hub defines a richer space |
| Contrastive vs other losses | InfoNCE binding is essential | The push/pull is what creates a shared, non-collapsed space |
| Learnable temperature | Helps the noisier modalities | Per-modality sharpness avoids over-punishing honest early errors |
| Projection head | Needed to map every modality to shared d | Without it, dimensions/scales don't match for a dot product |
| Augmentation strength | Stronger spatial aug helps depth/thermal | These modalities are small-data; aug fights overfitting |
Replacing the InfoNCE objective with a non-contrastive alternative degrades the emergent alignment. Without the negatives in the denominator, the space loses its spread (recall Ch. 3's collapse argument), and the triangle-inequality bridge of Ch. 4 has nothing to stand on — if everything collapses near one point, "near the anchor" stops being informative.
A crucial sanity check: adding a fifth or sixth modality to the joint training does not degrade the modalities already present. Each (modality, image) loss is essentially independent, sharing only the frozen image hub, so adding spokes is non-destructive. This is exactly what you want from a system meant to keep absorbing new senses.
From Ch. 4, emergent audio–text quality is bounded by α (audio–image angle) + β (text–image angle). Both angles are measured against the image embedding. If a stronger image encoder produces a more semantically faithful anchor, then both "dog-bark audio → dog image" and "dog text → dog image" become easier to satisfy tightly — shrinking both α and β at once, which tightens the bound on every emergent pair simultaneously. One improvement, global benefit. That's why the ablation singles out the anchor.
This is the payoff simulation. You control a tiny six-modality world. The image anchor sits at the center of the shared space. You decide how tightly each modality is bound to the image (the trained spokes), and the explorer computes, live, the full 6×6 cross-modal similarity matrix — including the emergent off-anchor pairs that were never trained. Watch the emergent cells light up as a pure consequence of the spokes you set.
Each slider tightens one modality's spoke to the image anchor (smaller angle = stronger binding). The grid is the live cosine-similarity matrix over all six modalities. The image row/column (highlighted) cells are trained; every other cell is emergent — it follows purely from the triangle-inequality geometry of the two spokes that touch it. Drive a spoke loose and watch every emergent cell that depends on it dim.
Notice the structure of the matrix. The image row and column are the only cells the loss ever optimized. Every other cell — text↔audio, depth↔thermal, audio↔IMU — is emergent. When you tighten two spokes, the emergent cell where they meet brightens, because the triangle bound angle(M₁, M₂) ≤ angle(M₁, img) + angle(M₂, img) shrinks. When you loosen a spoke, every emergent cell touching that modality dims at once. The image is doing all the binding from one place.
Play with the extremes. Tighten text and audio to near-zero angle: their emergent similarity approaches 1, even though it was never trained. Loosen IMU to 80°: every IMU-paired emergent cell collapses, while the others (which don't route through IMU) are untouched. This is the hub-and-spoke promise made tangible — independent spokes, one shared anchor, free cross-modal alignment that you can watch appear and disappear.
ImageBind is a beautiful idea with real edges. Knowing the limits is part of understanding the contribution.
Emergent ≠ trained. Cross-modal performance on unpaired pairs, while strong, is bounded by the triangle inequality and is generally below what direct (modality, modality) supervision would give. If you have real (audio, text) data, training on it directly will beat the emergent path.
The image is a bottleneck. Everything routes through the image anchor. Concepts that are hard to render visually — abstract ideas, fine-grained acoustic texture without a visual correlate — are poorly served, because their only bridge to other modalities is an image that may not capture them. A sound with no visual signature has a weak spoke.
Specialist models still win in-domain. A model trained end-to-end on a single modality with full supervision typically outperforms ImageBind on that modality's own benchmark. ImageBind trades peak per-modality accuracy for breadth and zero-shot transfer.
Bias and coverage. The space inherits the biases and gaps of the image–text pretraining and of each paired dataset. Modalities with little image-paired data (a small thermal corpus) get a noisier spoke and weaker emergent behavior.
Two bars per modality pair: the (hypothetical) accuracy if you trained the pair directly (teal) vs ImageBind's emergent accuracy through the image (warm). The gap is the price of routing through the anchor. Click to cycle which pair is shown; note the image-paired bars have no gap (they ARE trained).
ImageBind sits on top of a contrastive lineage and underneath a wave of any-to-any multimodal models. Follow these threads:
| Topic | Where | Why it connects |
|---|---|---|
| Contrastive CLIP | contrastive-clip.html | The InfoNCE loss and image–text binding ImageBind reuses and extends |
| Vector embeddings | vector-embeddings.html | The shared-space vectors and why a single dimension/space is required |
| Similarity metrics | similarity-metrics.html | Cosine similarity and L2-normalization — the geometry behind emergence |
| Multimodal RAG | multimodal-rag.html | Cross-modal retrieval over one embedding space in practice |
| Vision-Language Models | vlm.html | Downstream systems that can be upgraded with ImageBind's new senses |
| Transformers | attention-is-all-you-need.html | The encoder backbone every modality tower uses; the √d temperature analogy |
| Concept | One-line summary |
|---|---|
| Core idea | Bind 6 modalities into one space using only image-paired data |
| Modalities | image, text, audio, depth, thermal, IMU |
| Training pairs | Only (modality, image) — never (modality, modality) |
| Datasets needed | n−1 = 5 (vs 15 for the full pairwise mesh) |
| Loss | Symmetric InfoNCE (cross-entropy over scaled cosine sims), per modality, summed |
| Why cross-modal works | Triangle inequality: two vectors near a common image anchor are near each other |
| Anchor handling | Image & text encoders frozen; new-modality encoders + projection trained |
| Geometry | L2-normalized vectors on the unit sphere; similarity = cosine = dot product |
| Temperature | Learnable, per-modality; sharpens the contrastive softmax (~0.05) |
| Headline result | State-of-the-art emergent zero-shot on untrained modality pairs |
| Biggest lever | Image (hub) encoder capacity — improves every spoke at once |
| Main limit | Emergent quality bounded by the spokes; everything bottlenecks through the image |