OpenCLIP: Reproducible Scaling Laws for Contrastive Language-Image Learning

Chapter 1: The Dual-Encoder Architecture

CLIP, and therefore OpenCLIP, is two networks that never share weights but learn to agree. One reads pixels; one reads text. Each maps its input into the same d-dimensional vector space, and training pulls matching image-text pairs together while pushing mismatched pairs apart.

Concretely, trace one batch of B image-text pairs through the network. Let d be the shared embedding dimension (512 for ViT-B, 768 for ViT-L, 1024 for ViT-H/G).

The two crucial design decisions live in the last two rows. First, L2 normalization: every embedding is divided by its own length, so it lands on the unit hypersphere. After this, the dot product between an image vector and a text vector is exactly their cosine similarity — a number in [−1, 1] that depends only on the angle between them, not their magnitudes. The model can no longer cheat by making "confident" pairs simply longer vectors; it must align their directions.

Second, the similarity matrix. With B images and B texts, every image is scored against every text, producing a B×B grid. The diagonal entries are the B correct pairs (image i with its own caption i); the off-diagonal entries are the B²−B mismatched pairs. This single matrix is the entire substrate of the loss — and it is why a large batch matters so much, as we'll see.

A worked forward pass (tiny numbers)

Take d = 2 and B = 3 so we can do it by hand. Suppose after the encoders and L2 normalization we have:

(Each row already has unit norm: 1²+0² = 1, 0.6²+0.8² = 0.36+0.64 = 1, and so on.) The raw similarity matrix S = image_emb · text_emb^T has entry S_ij = cosine(image i, text j):

Read row 1: image 0 is most similar to text 0 (0.80) — good, that's the correct pair. Row 2: image 1 is most similar to text 0 (0.96), not its own text 1 (0.80) — a mistake the loss will punish. Row 3: image 2 is most similar to text 1 (1.00), not text 2 (0.80) — another mistake. An untrained model produces a messy S like this; training drives the largest value in each row onto the diagonal.

python
import torch, torch.nn.functional as F

# B image-text pairs through the two encoders
img_feat  = image_encoder(images)   # [B, d]
txt_feat  = text_encoder(tokens)    # [B, d]

# L2-normalize: every row becomes a unit vector on the hypersphere
img_emb = F.normalize(img_feat, dim=-1)  # [B, d]
txt_emb = F.normalize(txt_feat, dim=-1)  # [B, d]

# Cosine similarity of EVERY image with EVERY text
S = img_emb @ txt_emb.T            # [B, B], entries in [-1, 1]
# diagonal = the B correct pairs; off-diagonal = the B^2 - B negatives

Chapter 2: The Symmetric Contrastive Loss

Now the heart of the method — and the mechanism you'll implement in the Code Labs. We have the B×B similarity matrix S. We want to push every diagonal entry (correct pair) up and every off-diagonal entry (wrong pair) down. The tool for that is the InfoNCE loss (also called the contrastive loss), applied symmetrically across both axes.

Step 1: scale by the temperature

First we multiply S by a scalar called the logit scale. CLIP parameterizes it as exp(t) where t is learned, and we'll call τ the temperature with logit_scale = 1/τ. The scaled logits are:

Cosine similarities live in [−1, 1] — far too flat to act as softmax logits, which would give an almost-uniform distribution. Multiplying by a large factor (a trained value of 1/τ around 100) spreads them out so the softmax can become sharp. We devote all of Chapter 3 to why this scalar matters so much; for now treat it as a knob that sharpens the distribution.

Step 2: read each row as a classification problem

Here is the key reframing. Look at row i of L: it contains the score of image i against every one of the B texts. We want the highest score to be at column i (its own caption). That is exactly a B-way classification problem where the correct class is i. So we apply softmax across the row and take the cross-entropy against the target label i:

In words: for each image, turn its row of similarities into a probability distribution over the B texts, and penalize the model by the negative log-probability it assigns to the correct text. This is the image-to-text loss — "given this image, find its caption."

Step 3: do it again the other way (the "symmetric" part)

By exactly the same logic, each column j of L is a classification problem: "given this text, find its image," with correct answer j. Apply softmax down the columns (equivalently, over the rows of L^T) and cross-entropy against label j:

The final CLIP loss is the average of the two directions:

A fully worked example (B = 3, by hand)

Use the scaled logit matrix below (already multiplied by 1/τ). We'll compute the image-to-text loss completely.

Row 0 softmax: exponentials are e⁴=54.60, e¹=2.72, e⁰=1.00; sum = 58.32. So p_0→0 = 54.60/58.32 = 0.936. Loss contribution: −log(0.936) = 0.066.

Row 1: e²=7.39, e³=20.09, e¹=2.72; sum = 30.20. p_1→1 = 20.09/30.20 = 0.665. Contribution: −log(0.665) = 0.408.

Row 2: e⁰=1.00, e²=7.39, e⁵=148.41; sum = 156.80. p_2→2 = 148.41/156.80 = 0.947. Contribution: −log(0.947) = 0.055.

Average: loss^img = (0.066 + 0.408 + 0.055)/3 = 0.176. Row 1 dominates the loss because the model is least confident there (image 1 scores 2.0 on text 0, almost as high as 3.0 on its correct text 1). The gradient will push image 1's embedding away from text 0 and toward text 1. Doing the same column-wise gives loss^txt, and the reported loss is their average.

python
import torch, torch.nn.functional as F

def clip_loss(img_emb, txt_emb, logit_scale):
    # img_emb, txt_emb: [B, d], already L2-normalized
    logits = logit_scale * img_emb @ txt_emb.T   # [B, B]
    B = logits.shape[0]
    labels = torch.arange(B, device=logits.device)  # target = diagonal
    # image->text: each ROW is a B-way classification, answer = i
    loss_i = F.cross_entropy(logits, labels)
    # text->image: each COLUMN is a B-way classification, answer = j
    loss_t = F.cross_entropy(logits.T, labels)
    return (loss_i + loss_t) / 2

Notice how compact this is. The labels are simply arange(B) — the diagonal — because the i-th image's correct text is at index i by construction. The entire contrastive objective is two cross-entropies over a B×B matrix. That simplicity is what let it scale to billions of pairs.

Chapter 3: Temperature & the Power of a Big Batch

Two scalars dominate contrastive training in ways that surprise people the first time: the temperature τ and the batch size B. Both are about the same thing — how sharply the model is forced to discriminate — and OpenCLIP's scaling depended on getting both right.

Temperature: a learned sharpness knob

Recall the loss uses logits L = (1/τ)·S where S is cosine similarity in [−1, 1]. The factor 1/τ is the logit scale. If τ is large (logit scale small, say 1), the softmax over a row is nearly uniform — even the correct pair gets only a slightly higher probability, so the loss can't distinguish good from bad alignment. If τ is tiny (logit scale huge, say 1000), the softmax becomes one-hot — the model bets everything on its top guess, and a single mistake produces a gigantic loss, making training unstable.

CLIP learns τ rather than fixing it. It parameterizes logit_scale = exp(t) with t a trainable parameter, and clamps it so that logit_scale ≤ 100 (i.e. τ ≥ 0.01) to prevent it from collapsing to a degenerate one-hot regime. Empirically t converges so that 1/τ sits near 100 — a sweet spot where the softmax is sharp enough to separate the diagonal but smooth enough to keep gradients well-behaved.

Worked example: same matrix, two temperatures

Take a single row of cosine similarities S = [0.4, 0.1, 0.0] where index 0 is correct. At logit scale 5 the logits are [2.0, 0.5, 0.0]; softmax = [0.74, 0.17, 0.09], so −log(0.74) = 0.30. At logit scale 25 the logits are [10, 2.5, 0]; softmax ≈ [0.9994, 0.0006, 0.0001], so −log(0.9994) = 0.0006. Same embeddings, same cosines — but the higher scale makes the model far more confident, and therefore far more sensitive to any residual error. The gradient signal on the off-diagonal entries is proportionally larger, which is what drives faster separation early in training.

Batch size = number of negatives

The other lever is B. Each image is contrasted against the B−1 other texts in its batch. A larger batch gives each image a harder, more informative discrimination task: it must stand out not against 255 distractors but against 32,767. The expected loss of pure guessing is log(B), so as B grows the model is solving a strictly harder classification, and the representations it learns are correspondingly more discriminative.

This is why CLIP-style training uses enormous global batch sizes — 32k and beyond — assembled across many GPUs. OpenCLIP pushed batch sizes to tens of thousands and found this essential for the largest models. The catch: the B×B logit matrix and the all-gather of embeddings across GPUs make very large batches a systems-engineering problem, not just a hyperparameter.

python
import torch, torch.nn as nn, numpy as np

# logit_scale is a learned parameter, initialized to log(1/0.07) ~ 2.66
logit_scale = nn.Parameter(torch.tensor(np.log(1/0.07)))

# in the forward pass, exponentiate and clamp so 1/tau never exceeds 100
scale = logit_scale.exp().clamp(max=100.0)
logits = scale * img_emb @ txt_emb.T   # [B, B]

Chapter 4: Zero-Shot Transfer — the Evaluation

The whole point of pre-training is what you can do without further training. CLIP turns classification into retrieval, and that conversion is worth understanding precisely because it is how every number in the scaling-law plots was measured.

How a contrastive model classifies

To classify an image among C candidate classes (say the 1000 ImageNet classes), you do not add a head. Instead:

The class text embeddings act as a weightless classifier head: you compute them once and reuse them for the whole test set. Swapping to a different dataset means swapping the prompts, not retraining. This is precisely what makes a single pre-trained OpenCLIP model directly comparable across dozens of downstream tasks — and therefore what makes a clean scaling law possible.

Prompt engineering and ensembling

The exact wording of the prompt matters. "a photo of a {c}" beats the bare class name "{c}" because the pre-training captions were natural sentences, not isolated words — the prompt nudges the text embedding into the same distribution the model saw during training. CLIP further uses prompt ensembling: embed each class under many templates ("a photo of a {c}", "a blurry photo of a {c}", "a sculpture of a {c}", …), then average the resulting text vectors before normalizing. Averaging cancels template-specific noise and reliably adds a point or two of accuracy.

Worked example: a 3-class decision

Image vector v = [0.6, 0.8] (already normalized). Three class embeddings: cat = [0.7, 0.7], dog = [0.0, 1.0], car = [−0.9, 0.4] (each near unit norm). Cosine scores: v·cat = 0.6(0.7)+0.8(0.7) = 0.42+0.56 = 0.98; v·dog = 0+0.8 = 0.80; v·car = −0.54+0.32 = −0.22. The argmax is cat (0.98). Apply softmax with logit scale 100 and the model is essentially certain. No ImageNet labels were ever touched — the decision came entirely from the shared embedding space.

python
import torch, torch.nn.functional as F

# Build the weightless classifier head once
classes = ["cat", "dog", "car"]
prompts = [f"a photo of a {c}" for c in classes]
text_emb = F.normalize(text_encoder(tokenize(prompts)), dim=-1)  # [C, d]

# Classify a new image with no training
img_emb = F.normalize(image_encoder(image), dim=-1)   # [1, d]
scores  = (img_emb @ text_emb.T)                       # [1, C] cosines
pred    = classes[scores.argmax()]                   # nearest caption

Chapter 5: Scaling Laws for Contrastive Learning

Here is the paper's central scientific contribution. Once you can train the same recipe at many scales, you can ask: how does downstream accuracy depend on how much you spend? The answer, across more than two orders of magnitude, is a power law.

The shape of the law

Let C be the total training compute (roughly: model size × number of samples seen), and let E be the downstream error (1 − accuracy). The empirical finding is that error falls as a power of compute:

Three pieces. β sets the overall height. The exponent α (a small positive number) is the slope on a log-log plot — it says how fast error drops as you add compute. E_∞ is an irreducible floor: the error you'd have with infinite compute, set by label noise and the intrinsic difficulty of the task. On a log-log axis, the power-law part is a straight line with slope −α, which is why these plots look so clean and so predictive.

The crucial caveat: data distribution sets the exponent

The most cited finding of the paper is not that scaling laws exist — Kaplan and Hoffmann had shown that for language models. It is that the scaling exponent depends on the training data distribution. OpenCLIP found that models trained on LAION (open) and models trained on WIT (OpenAI's closed set) scale along different power laws, and which one is "better" flips depending on the downstream task. Roughly: OpenAI's CLIP had a steeper improvement on ImageNet-style classification, while LAION-trained models scaled better on retrieval tasks. The data, not just the model, controls the slope.

This is a deep point. It means you cannot speak of "the" scaling law for contrastive learning. There is a scaling law for a given dataset evaluated on a given task. Bigger models help, more data helps, more compute helps — but the rate of help is a property of the data distribution you train on. Choosing the dataset is choosing your slope.

Bottleneck analysis: which axis is starved?

Scaling has three axes — model size, dataset size, and samples seen (compute) — and a model can be bottlenecked on any one. If you grow the model but freeze the data, accuracy plateaus: you've saturated what the data can teach. If you grow the data but freeze the model, the model can't absorb it. The paper's grid lets you read off which axis is the binding constraint at each scale, and the practical guidance is to grow all three roughly in concert.

Chapter 6: Data Scale, Curation & Deduplication

A scaling study is only as trustworthy as its data. If your test images leak into your training set, every accuracy number is inflated. OpenCLIP put real engineering into building LAION and into proving the scaling wasn't an artifact of contamination.

How LAION was built

LAION (Large-scale Artificial Intelligence Open Network) datasets are filtered from Common Crawl. The pipeline: extract every <img> tag with a non-empty alt attribute (the alt-text is the "caption"), then filter by CLIP itself — keep only pairs whose existing-CLIP cosine similarity exceeds a threshold (around 0.28 for the English subset). This bootstraps quality from a weaker model: pairs where the image and its alt-text already roughly agree are kept; random or broken alt-text is dropped. The result is LAION-400M, then LAION-2B (the English subset of the multilingual LAION-5B).

Why deduplication is non-negotiable

Common Crawl is full of near-duplicate images, and some of them are the very ImageNet validation images you evaluate on. If "test" images sit in "train," zero-shot accuracy is no longer zero-shot — the model has effectively seen the answer. OpenCLIP therefore ran deduplication: compute a perceptual/embedding hash of every training image, find near-duplicates of the evaluation sets, and remove them. The critical experiment is to retrain with the overlap removed and confirm the accuracy barely moves. It did, which means the scaling gains are real generalization, not memorized test images.

Worked example: how much can dedup matter?

Suppose 1.0% of your training set are near-duplicates of test images, and the model gets those "seen" examples right 99% of the time vs. 75% for genuinely novel ones. The inflation to a measured 76% accuracy is approximately 0.01×(0.99−0.75) = 0.0024, i.e. about a quarter of a point — small here, but it grows with the contamination fraction and is exactly the kind of effect that can manufacture a fake "scaling" trend if it correlates with dataset size. The only way to know it's small is to measure it by retraining deduplicated, which is what the paper did.

python
# LAION-style filtering: keep pairs an existing CLIP already finds agreeable
import torch, torch.nn.functional as F

img_emb = F.normalize(clip_image(image), dim=-1)
txt_emb = F.normalize(clip_text(alt_text), dim=-1)
sim     = (img_emb * txt_emb).sum(-1)        # cosine
keep    = sim > 0.28                          # English-subset threshold

# Dedup: drop training images that near-match any eval image
h_train = perceptual_hash(image)
contaminated = h_train in eval_hash_set      # remove before training

Chapter 7: Headline Results & Ablations

With the recipe, the loss, and the data understood, what did the runs actually show? The numbers below are the load-bearing claims of the paper.

The flagship number

An OpenCLIP ViT-G/14 trained on LAION-2B reached roughly 80% zero-shot top-1 on ImageNet. This is the result that mattered most politically: an entirely open pipeline — open data, open code, open weights — matched and surpassed OpenAI's closed CLIP (whose released ViT-L/14 sat in the high-70s). The open ecosystem was no longer a second-class imitation; it was the frontier.

(These figures are approximate and depend on training duration; treat them as the shape of the ladder, not exact decimals.) The point is the monotone climb: each rung — more parameters, more samples seen, more data — buys a predictable increment of accuracy, exactly as the power law promises.

Beyond ImageNet: robustness and transfer

The paper evaluated zero-shot across dozens of datasets, not just ImageNet. Two patterns stand out. First, robustness: OpenCLIP models held up well on distribution-shifted ImageNet variants (ImageNet-R, ImageNet-Sketch, ObjectNet), where supervised models often collapse — a known strength of contrastive pre-training on diverse web data. Second, task-dependence of scale: the relative ranking of LAION- vs WIT-trained models flipped between classification and retrieval, the empirical core of the data-distribution finding from Chapter 5.

What the ablations isolated

The careful, reproducible grid let the authors attribute gains to specific axes rather than to a lucky run:

Chapter 8: The Scaling Explorer

This is the payoff — the paper's central mechanism made fully interactive. Train a (simulated) OpenCLIP by choosing a model size, a dataset, and a compute budget. The explorer computes the implied zero-shot accuracy from a power law whose exponent depends on the dataset you pick — the paper's key finding — and shows you the contrastive batch the model is learning from underneath.

Watch two things as you slide the controls. (1) The accuracy gauge climbs along the power law; doubling compute moves you a fixed step along the log-log line, not a fixed number of points. (2) The similarity matrix on the right sharpens its diagonal as effective training increases — more negatives and more compute make the model better at putting each image's caption on the diagonal. (3) Switch the dataset and the slope changes: the same compute buys a different accuracy because the data distribution sets the exponent.

The explorer encodes the three lessons of the paper in one picture. Model size shifts which curve you're on. Dataset choice tilts the curve's slope (the exponent). Compute slides you along it. And the matrix reminds you what is actually being optimized the whole time: a symmetric contrastive objective pushing the diagonal up and everything else down. Scale is not magic — it is moving along a measured line, on a slope your data picked.

Chapter 9: Limitations & Connections

OpenCLIP changed what the open community could do, but it inherits real limits — and pointing them out honestly is part of understanding the contribution.

Limitations

Where it led

OpenCLIP became the default vision backbone for the open generative era. Stable Diffusion's text conditioning uses an OpenCLIP text encoder; countless retrieval, captioning, and VLM systems build on OpenCLIP image features. The open_clip library and the LAION datasets turned "train your own CLIP" from an OpenAI-only capability into a community one. The data-distribution finding directly motivated later data-centric work (DataComp, MetaCLIP) that scales the dataset curation rather than just the model.

Connections on Engineermaxxing

The cheat sheet