How neural networks learn what "similar" means, and how CLIP taught machines to see through the lens of language.
Raw data — pixels, audio samples, characters — is not directly useful for reasoning. A 224×224 image is 150,528 numbers, but those numbers don't tell you "this is a dog" or "this looks like a Van Gogh painting." A representation is a transformation that turns raw data into useful numbers.
A good representation puts similar things close together and different things far apart in an embedding space. A photo of a golden retriever and a labrador should have nearby embeddings. A golden retriever and a car should be far apart.
Points are data samples. Colors are categories. A good representation clusters similar items together. Click "Shuffle" to see a bad (random) embedding; click "Organize" to see a learned one.
Contrastive learning is the art of learning embeddings by comparing pairs. For each sample (the anchor), you need a positive (something similar) and negatives (things that are different). The loss pulls anchor and positive together and pushes anchor and negatives apart.
The beauty: you don't need class labels. The "positive" can be a different augmentation of the same image: flip it, crop it, change the color. Any two views of the same image are a positive pair. Everything else in the batch is a negative.
The anchor is pulled toward the positive and pushed away from negatives. Click "Step" to see the forces applied. Watch embeddings organize over steps.
SimCLR (Simple Contrastive Learning of Representations) showed how effective contrastive learning can be with a clean, minimal design. Take a batch of N images. Augment each twice to get 2N views. For each image, its two augmented views form the positive pair. The other 2(N-1) views are negatives.
The architecture: a ResNet encoder maps each view to an embedding, then a small projection head (MLP) maps it to the space where the contrastive loss is applied. After training, the projection head is thrown away — the encoder's output is the useful representation.
| Augmentation | What it does | Why it helps |
|---|---|---|
| Random crop + resize | Different spatial views | Forces spatial invariance |
| Color jitter | Adjust brightness, contrast, saturation | Prevents color-based shortcuts |
| Horizontal flip | Mirror the image | Orientation invariance |
| Gaussian blur | Smooth the image | Prevents texture-based shortcuts |
h with dimension 2048. A two-layer MLP maps it to z with dimension 128. The contrastive loss is computed on z, while downstream linear evaluation uses h.The workhorse of contrastive learning: an InfoNCE-style loss (SimCLR calls its normalized, temperature-scaled variant NT-Xent). For paired banks indexed so that ki matches qi, the loss is the negative log probability assigned to that paired key among all candidates in the denominator. It's essentially a softmax over similarity scores.
In the SimCLR variant, both views live in one 2N-view bank: exclude the anchor itself, include its paired view once, and treat the other 2N−2 views as competitors. The lab below uses separate query and key banks, so the anchor is not itself a candidate.
The temperature τ controls softmax sharpness. Low τ concentrates probability and gradient weight on the highest-scoring candidates; high τ spreads weight more evenly, approaching a uniform distribution only as τ approaches infinity. The similarity function is cosine similarity: sim(a,b) = a·b / (||a|| ||b||).
exp(logit_scale) = 1 / tau, initialized to 1/0.07.Two paired banks—B image embeddings and B text embeddings—create a B×B matrix. The diagonal marks paired positives; training aims to score them above the off-diagonal alternatives. Adjust temperature to see the row-softmax effect.
Under CPC's candidate-sampling assumptions, one positive comes from the joint distribution and the other N−1 candidates are independent draws from the marginal. Define LN as the expected negative log probability assigned to the positive. CPC establishes I(X;Y) ≥ log N − LN.
Your task: Rearrange the inequality and explain what increasing N changes. Use natural logs; divide by log 2 only when reporting bits.
CPC gives I(X;Y) ≥ log N − LN. Lowering the loss raises this bound; increasing N raises its ceiling and can tighten it. The result does not prove that a model with batch size N can represent only log N nats, nor that CLIP's batch size is theoretically required.
Let ak = sim(zi,zk)/τ and p = softmax(a). Derive the anchor gradient and identify both places where τ acts.
∇z_iL = (1/τ)[∑kpk∇sim(zi,zk) − ∇sim(zi,zpos)]. A smaller τ changes the explicit prefactor and also changes every pk, concentrating relative weight on high-similarity candidates. It does not guarantee that every gradient coordinate grows.
Assume a row already ranks the positive above every negative. What happens if every raw dot-product logit is multiplied by a growing positive scale?
The softmax can become more confident without changing the ordering. L2-normalizing each projected feature removes per-example norm from the score, turning the dot product into cosine similarity. CLIP then uses one learned global logit scale. If the positive is not already ranked above a negative, scale alone cannot repair that error.
Both form softmax weights from scaled dot products. In the contrastive loss, a known paired index supplies a classification target. In attention, the weights combine value vectors into a contextual representation. The algebraic weight pattern is similar, but the weights serve different operations.
Why are these scales not interchangeable? Attention divides by √d to compensate for dimension-dependent dot-product variance; contrastive τ calibrates the sharpness of similarity logits. Both rescale logits before softmax, but they solve different problems.
For a naive alignment loss such as one minus cosine similarity, a constant vector gives every positive pair perfect agreement. SimCLR adds in-batch negatives, so mapping different samples to the same point is penalized. Negatives are one anti-collapse mechanism; DINO uses centering, sharpening, and a momentum teacher instead.
CLIP (Contrastive Language-Image Pre-training) applies contrastive learning across modalities: images and text. Instead of two augmented views of the same image, the positive pair is an image and its caption. Train on 400 million image-text pairs from the internet.
Two separate encoders: a vision encoder (ViT or ResNet) maps images to embeddings, and a text encoder (Transformer) maps captions to embeddings. The contrastive loss pulls matching (image, caption) pairs together and pushes non-matching pairs apart.
Let's trace the exact data flow. The image path: a batch of images [batch, 3, 224, 224] enters ViT-L/14. The image is split into 14×14-pixel patches — a 16×16 grid at 224×224 — each flattened and projected to 1024 dims, giving 256 patch tokens plus one class token — 257 tokens total. After 24 transformer layers, the class token (a single 1024-dim vector) is extracted and passed through a linear projection head that maps it to 768 dimensions. The result is L2-normalized: [batch, 768].
The text path: a caption is tokenized with BPE (byte-pair encoding, 49,152-token vocabulary), padded/truncated to 77 tokens. A 12-layer transformer processes these tokens. The embedding at the [EOS] token position (analogous to the class token) is extracted and projected to 768 dimensions. Also L2-normalized: [batch, 768].
Images (squares) and text (circles) in a shared space. Matching pairs are connected. CLIP learns to align them.
CLIP's training is elegant. Take a batch of N (image, text) pairs. Compute all N×N cosine similarities. The diagonal contains matching pairs. Apply cross-entropy loss to make each row and column peak at the diagonal. That's it.
In code, this is shockingly simple. You have image_embeds of shape [N, 768] and text_embeds of shape [N, 768], both L2-normalized. The similarity matrix is one matrix multiply:
This gives an [N, N] matrix. The diagonal entries are the N positive pairs (imagei matched with texti). The off-diagonal entries are the N²−N negative pairs. For a concrete example with N=4: you get 4 positive pairs and 12 negatives. The loss is symmetric cross-entropy — cross-entropy on the rows (which text matches each image?) AND on the columns (which image matches each text?), averaged.
The authors constructed WIT from 400 million image-text pairs collected from publicly available internet sources. Their largest Vision Transformer trained for 12 days on 256 V100 GPUs with a batch size of 32,768. The batch supplies many in-batch alternatives, but the paper does not establish that this batch size is theoretically required.
Each cell is the cosine similarity between image i and text j. The loss tries to make the diagonal bright (high similarity) and everything else dark (low similarity).
| Hyperparameter | Value | Why |
|---|---|---|
| Batch size | 32,768 | Many in-batch candidates in the reported setup |
| Temperature τ | 0.07-equivalent initialization; learned thereafter | Controls the row/column softmax sharpness |
| Image encoder | ViT-L/14 | Large Vision Transformer for best quality |
| Text encoder | 12-layer Transformer | Standard text encoding |
| Embedding dim | 512 or 768 | Shared dimension for image-text space |
exp(logit_scale) is the positive inverse temperature, 1/τ; multiplying by it is equivalent to dividing cosine similarities by τ.python import torch import torch.nn.functional as F def clip_loss(image_embeds, text_embeds, logit_scale): # Inverse temperature scale (learned, initialized to 1/0.07 ≈ 14.3) scale = torch.exp(logit_scale) # Similarity matrix: (N, N) logits = image_embeds @ text_embeds.T * scale # Labels: diagonal is the correct match N = logits.shape[0] labels = torch.arange(N, device=logits.device) # Symmetric cross-entropy loss_i2t = F.cross_entropy(logits, labels) # rows loss_t2i = F.cross_entropy(logits.T, labels) # cols return (loss_i2t + loss_t2i) / 2
Changing temperature, normalization, or the candidate set changes the optimization problem. Predict the qualitative score changes, run a controlled experiment, and limit the conclusion to the measured result.
The authors sharded the similarity calculation. Each GPU computed only the subset of pairwise similarities required for its local batch. The paper does not report every GPU storing the complete matrix, nor the 48 MB/96 MB accounting previously shown here.
CLIP's superpower: zero-shot classification. To classify an image, create text prompts for each class: "a photo of a cat", "a photo of a dog", "a photo of a car." Embed all prompts. Embed the image. Pick the text with highest cosine similarity. No training on the target task.
This works because CLIP's shared embedding space aligns concepts across modalities. "A photo of a dog" is near actual dog photos, even if CLIP never saw this specific classification task during training.
For the ViT-L/14 configuration, zero-shot ImageNet classification starts with all 1,000 ImageNet class names. Create a text template for each: "a photo of a {class}". Run them through the text encoder to get 1,000 embeddings of shape [1000, 768]; these can be computed once and reused across the image set. Encode an image to [1, 768], take its dot product with all 1,000 normalized text embeddings, and choose the argmax. No ImageNet-specific fitting is performed in this zero-shot evaluation.
An image is compared against text prompts for 5 classes. The bar chart shows similarity scores. The highest score wins. Click to generate a new random scenario.
SigLIP replaces CLIP's globally normalized softmax objective with independent sigmoid losses for image-text pairs.
This removes global normalization across a row or column. The paper reports better results at smaller batches and tests batches up to one million examples, but gains diminish quickly and 32K is sufficient in its reported study. Its million-example implementation still exchanges features through chunked collective permutes.
DINO trains a student to match the output distribution of a momentum teacher across augmented crops. Centering and teacher-output sharpening counter different collapse modes.
The paper finds that the last-layer class-token attention maps contain information about semantic regions and object boundaries, despite not being optimized as segmentation masks. It evaluates this structure with thresholded attention masks and video instance-segmentation transfer. These results do not alone establish generic robotics or depth claims.
Later systems reuse pretrained encoders in specific, documented ways. Original LLaVA freezes a CLIP ViT-L/14 visual encoder, takes its grid features, and trains a linear projection into the language model's word-embedding space. At 224 pixels, ViT-L/14 has a 16 by 16 grid: 256 patch features, each at the vision width of 1024 before projection.
Stable Diffusion v1 uses a frozen CLIP ViT-L/14 text encoder. Its non-pooled text representation conditions the denoising U-Net through cross-attention.
GPT-4V's public system card says the model accepts image inputs, but its vision-encoder architecture is not disclosed.