CLIP's softmax contrastive loss couples every image to every text in the batch through a global normalization. SigLIP throws that away — it scores each image-text pair independently with a plain sigmoid, decoupling the loss, simplifying the implementation, and winning at small batch sizes.
You want to train a CLIP-style model — an image encoder and a text encoder that map photos and captions into the same vector space, so that a picture of a corgi lands next to the words "a photo of a corgi." The standard recipe (Radford et al., 2021) is a contrastive loss: take a batch of n image-text pairs, compute the n × n matrix of image-text similarities, and train so the n matched pairs on the diagonal score high while the n² − n mismatched off-diagonal pairs score low.
Here is the catch that nobody puts on the poster. CLIP's loss is a softmax over each row and each column of that similarity matrix. A softmax has a denominator, and that denominator is a sum over every other entry in the row. To compute the loss for a single image, you need the similarity of that image to every caption in the batch. The loss is not a property of one pair — it is a property of the whole batch at once.
When the batch is split across, say, 256 accelerator chips for data parallelism, each chip holds only a few image-text pairs. But the softmax denominator needs all of them. So every step you must all-gather the full set of embeddings (or the full similarity matrix) across all chips, twice — once for the image-to-text softmax, once for the text-to-image softmax. That is a global synchronization barrier and an O(n²) communication cost, baked into the loss itself.
Top: the softmax loss for image 0 (warm) reaches across to every caption in the batch — that whole row is its denominator. Click "Animate" to watch the dependency sweep. Bottom: the sigmoid loss for the same pair touches only the single matched entry plus its own independent negatives. Notice the softmax needs the entire row before it can produce a single number.
| Property | CLIP (softmax contrastive) | SigLIP (sigmoid) |
|---|---|---|
| Loss granularity | Per row/column — needs whole batch | Per pair — fully independent |
| Normalization | Global softmax over all candidates | None — each pair is its own σ |
| Cross-device comms | Two all-gathers per step (O(n²)) | Cheap pairwise sums (O(n²) compute, no global softmax) |
| Small batch | Weak — few negatives, noisy denominator | Strong — outperforms softmax |
| Implementation | Symmetric two-way softmax + sync | A few lines of sigmoid cross-entropy |
Two things. First, the engineering pain above: the softmax forces a global reduction, so scaling the batch means scaling expensive cross-device communication, and small-batch training is left in the cold. Second, a subtler statistical point. The softmax normalizes over the negatives, so its gradient signal per pair is entangled with however many negatives happen to be in the batch this step. SigLIP decouples the positive term from the negative terms entirely — each pair gets its own loss whose scale you control directly with a learnable temperature and bias, not implicitly through the batch size.
By the end of this chapter you will have run, in your browser, the exact two losses on the same similarity matrix and watched them produce different gradients. Let us first make sure the CLIP baseline is crisp.
To appreciate what SigLIP changes, we need the CLIP loss in our hands, exactly. Suppose a mini-batch of n image-text pairs. An image encoder f turns image i into a unit vector xi; a text encoder g turns caption j into a unit vector yj. Both live in the same d-dimensional space. (In the paper, image embeddings have shape [n, d] and text embeddings [n, d], with d typically 768 or 1152.)
The logit for pair (i, j) is their scaled dot product:
where xi · yj is the cosine similarity (both vectors are L2-normalized) and t > 0 is a learnable temperature (CLIP parametrizes it as exp of a learnable scalar, clamped to avoid blow-up). Stack these into an n × n matrix of logits. The diagonal entries ℓii are the matched pairs; everything off-diagonal is a mismatch.
CLIP applies a softmax cross-entropy in both directions and averages. Reading along row i (image i against all n captions), the correct class is the diagonal:
Reading down column j (caption j against all n images), the correct class is again the diagonal:
The asymmetry between the two directions is the whole reason CLIP needs two softmaxes: the image-to-text denominator sums over a row, the text-to-image denominator sums over a column. Each is a different normalization of the same matrix.
Take n = 2 with the scaled logit matrix ℓ = [[4, 1], [0, 3]] (so ℓ00 = 4 and ℓ11 = 3 are the matched pairs, already including t). The image-to-text softmax for row 0 has denominator e4 + e1 = 54.60 + 2.72 = 57.32, so the matched probability is 54.60 / 57.32 = 0.953, and −log(0.953) = 0.049. Row 1 has denominator e0 + e3 = 1 + 20.09 = 21.09, matched prob 20.09 / 21.09 = 0.953, loss 0.049. So Limg→txt = (0.049 + 0.049)/2 = 0.049.
Now the columns. Column 0 has denominator e4 + e0 = 55.60, matched prob 54.60/55.60 = 0.982, loss 0.018. Column 1 has e1 + e3 = 22.81, matched prob 20.09/22.81 = 0.881, loss 0.127. So Ltxt→img = (0.018 + 0.127)/2 = 0.073. The total CLIP loss is (0.049 + 0.073)/2 = 0.061. Notice every one of those four denominators mixed entries from across the matrix — that mixing is exactly the coupling SigLIP will sever.
A 5×5 image-text similarity matrix (diagonal = matched pairs, warm). Toggle which softmax direction is active. The highlighted band is the set of entries that go into one softmax denominator — the whole row (image→text) or the whole column (text→image). The loss for one pair is never local.
Hold onto the matrix picture. Both losses we compare in this lesson act on the very same n × n logit matrix ℓ = t · XYT. They differ only in how they turn that matrix into a scalar.
Here is SigLIP's entire idea. Forget "which caption out of n." Instead ask, for every entry of the n × n matrix independently: is this image and this text an actual pair, yes or no? That is a binary classification, and the loss for binary classification is the sigmoid (logistic) cross-entropy.
Define a label zij that is +1 if i and j are a true pair (the diagonal, i = j) and −1 otherwise (every off-diagonal mismatch). The logit for the pair, exactly as in CLIP, is ℓij = t · xi · yj + b, where t is the learnable temperature and b is a new learnable bias (Chapter 3 explains why we suddenly need a bias). The sigmoid loss is:
Read the argument carefully. The quantity inside is zij · ℓij. For a positive pair z = +1, so we maximize σ(ℓij) — we want the matched logit high. For a negative pair z = −1, so we maximize σ(−ℓij) — we want the mismatched logit low. Each (i, j) cell contributes its own term, summed over the whole matrix. There is no denominator that mixes cells.
The standard binary cross-entropy with a ±1 label and logit ℓ is −log σ(z · ℓ). When z = +1 it is the "softplus" −log σ(ℓ) = log(1 + e−ℓ); when z = −1 it is log(1 + e+ℓ). So the SigLIP loss is just n² independent logistic regressions sharing the same encoders and the same (t, b). The diagonal supplies n positive examples; the off-diagonal supplies n² − n negatives.
Use the raw similarity matrix s = [[4, 1], [0, 3]] (we will fold t and b into these for the example, so ℓ = s here). The labels are z = [[+1, −1], [−1, +1]]. Now compute z · ℓ for each cell: z00ℓ00 = +4, z01ℓ01 = −1, z10ℓ10 = 0, z11ℓ11 = +3.
The per-cell loss is log(1 + e−(zℓ)). Cell (0,0): log(1 + e−4) = log(1.0183) = 0.0181. Cell (0,1): log(1 + e+1) = log(3.718) = 1.313. Cell (1,0): log(1 + e0) = log(2) = 0.693. Cell (1,1): log(1 + e−3) = log(1.0498) = 0.0486. Sum = 2.073, divide by n = 2: Lsig = 1.036.
Compare to the CLIP loss on the same matrix from Chapter 1: 0.061. They are completely different numbers, and — more importantly — their gradients point in different directions. The sigmoid is screaming about cell (0,1) and (1,0) (the easy-to-improve negatives) in a way the softmax, with its competitive budget, partly masks. This is not a bug; it is the different inductive bias.
The same 5×5 similarity matrix, now scored by the sigmoid loss. Each cell is shaded by its own loss value (brighter = higher loss), with NO reference to any other cell. Diagonal cells (positives, ringed warm) want a high logit; off-diagonal cells (negatives) want a low logit. Drag the slider to move every logit up or down and watch positives and negatives trade off.
python # SigLIP loss vs CLIP loss on the SAME similarity matrix import numpy as np def sigmoid(z): return 1.0 / (1.0 + np.exp(-z)) # n image-text pairs, already L2-normalized embeddings # logits[i,j] = t * (x_i . y_j) + b (t,b folded in below) logits = np.array([[4., 1.], [0., 3.]]) n = logits.shape[0] # z = +1 on the diagonal (true pairs), -1 off-diagonal z = 2.0 * np.eye(n) - 1.0 # [[1,-1],[-1,1]] # SigLIP: independent sigmoid per cell, averaged L_sig = -np.mean(np.log(sigmoid(z * logits)).sum(axis=1)) # L_sig = 1.036 # CLIP: two-way softmax cross-entropy, averaged def ce(M): lse = np.log(np.exp(M).sum(axis=1)) # row log-sum-exp return np.mean(lse - np.diag(M)) # -log p(diagonal) L_clip = 0.5 * (ce(logits) + ce(logits.T)) # L_clip = 0.061 -- a different number AND a different gradient
The sigmoid loss as written would fail catastrophically at the start of training, and the reason is a counting problem. Look at the matrix: there are n positives on the diagonal and n² − n negatives off it. For a batch of 16,384, that is 16,384 positives against roughly 268 million negatives. At initialization the encoders are random, so every logit is near zero, every σ is near 0.5, and the negative term −log σ(−ℓ) ≈ −log(0.5) is paid 16,000 times over for every single positive.
That overwhelming negative mass produces a huge initial gradient that simply shoves all logits as negative as possible — positives included. The model spends the early steps just learning "everything is a mismatch," which is a terrible, slow place to start. SigLIP's fix is a single learnable scalar.
Add a learnable bias b to every logit: ℓij = t · xi · yj + b. Initialize b to a large negative value (the paper uses binit = −10). With b = −10 at the start, every logit is around −10, so σ(−ℓ) for a negative is σ(10) ≈ 0.99995 — the loss on each negative is essentially zero. The model starts from the sensible prior "most pairs are mismatches" (which, in a random batch, is true: only the diagonal matches), so the early gradient is dominated by the few positives it needs to pull up rather than the flood of negatives it would otherwise fight.
This is exactly the trick used to stabilize one-vs-all detectors with extreme class imbalance (the "prior bias" of focal-loss-era object detectors): bias the classifier toward the majority class at init so the rare positives are not drowned.
The temperature t scales the cosine similarity (which lives in [−1, 1]) up into a useful logit range. Like CLIP, SigLIP parametrizes t' = log t as the free variable and uses t = exp(t'), so t stays positive, and initializes t' = log 10 (so t starts at 10). As training proceeds the model learns how sharp it wants the decision boundary: a larger t makes σ steeper, so small cosine differences become confident yes/no decisions.
Take a tiny n = 4 batch at init with all cosine similarities ≈ 0 (random encoders) and t = 10, so the t-scaled term is ≈ 0. With b = 0: every logit is 0, every σ = 0.5. The total negative loss is (n² − n) · −log(0.5) = 12 · 0.693 = 8.32, while the positive loss is n · −log(0.5) = 4 · 0.693 = 2.77. Negatives outweigh positives 3:1 even at this toy size; at batch 16k it is 16,000:1.
Now with b = −10: every logit is −10. Negative loss per cell is −log σ(10) = −log(0.99995) = 0.000045, times 12 = 0.00055 — negligible. Positive loss per cell is −log σ(−10) = −log(0.000045) = 10.0, times 4 = 40.0. The loss landscape is now driven almost entirely by the positives the model must learn to recognize, exactly as desired. The model climbs out of the −10 prior only for genuine matches.
Drag the initial bias b. The two bars show the total loss mass coming from the n positives (warm) vs the n²−n negatives (teal) at initialization, for a batch of n shown by the second slider. Without a strongly negative bias, the negatives dominate by orders of magnitude; the paper's b = −10 flips control to the positives.
python import torch import torch.nn as nn class SigLIPHead(nn.Module): def __init__(self): super().__init__() # learnable log-temperature, init so t = exp(log 10) = 10 self.t_prime = nn.Parameter(torch.tensor(2.3026)) # log(10) # learnable bias, init -10 to counter negative-pair flood self.b = nn.Parameter(torch.tensor(-10.0)) def forward(self, img_emb, txt_emb): # img_emb, txt_emb: [n, d], already L2-normalized t = self.t_prime.exp() # positive temperature logits = t * img_emb @ txt_emb.T + self.b # [n, n] return logits
The single biggest systems payoff of the sigmoid loss is how it parallelizes. Because each cell is independent, the loss decomposes cleanly across devices, and the all-gather of the full batch that the softmax demanded simply disappears.
Suppose data parallelism over D devices, each holding a local chunk of b = n/D pairs. The softmax denominator for any image needs all n captions. So each device must receive the embeddings of every other device's text (and images) — an all-gather of the full [n, d] embedding tensors, every step, in both directions. Communication grows with the batch, and the loss cannot even start until the gather completes (a hard barrier).
SigLIP computes the sigmoid loss as a sum over the matrix, and a sum can be reorganized however you like. The paper's trick: each device first computes the loss for its own diagonal block (its local images vs its local texts — including the positives). Then the devices pass their text representations around the ring of D devices, one hop at a time. On each hop, every device computes the loss between its local images and the incoming texts — all negatives — and adds it in. After D − 1 hops every image has seen every text, and the full loss is accumulated.
The key property that makes this legal is that the sigmoid loss is a plain sum of per-cell terms with no shared denominator. You could never reorganize a softmax this way — each row's normalizer needs the whole row simultaneously before any term is correct. The sigmoid's independence is what unlocks the memory-flat, communication-light implementation.
Four devices, each owning one block of images (rows) and texts (columns). Click "Hop" to pass texts one step around the ring. The cells covered so far light up; the diagonal blocks (positives) are computed locally at step 0 with zero communication. After 3 hops the full matrix is covered — no device ever held more than two blocks at once.
The headline empirical finding of the paper is about batch size, and it is genuinely surprising. Conventional CLIP wisdom said "bigger batch is always better" — more negatives in the softmax denominator means a harder, more informative contrast. SigLIP complicates that story in two directions.
When the batch is small (a few thousand or fewer), the softmax denominator has few candidates, so its normalization is noisy and the loss is weak. The sigmoid loss does not depend on a denominator at all — each pair's loss is well-defined regardless of how many other pairs are present. Across the small-to-moderate batch regime, SigLIP trains better than the equivalent softmax CLIP. This is the result that matters for practitioners who do not own thousands of accelerators.
As the batch grows into the tens of thousands, the softmax's denominator becomes rich and the gap between the two losses narrows; at very large batch the two perform comparably. More striking: both losses saturate. The paper finds performance keeps improving up to a batch of roughly 32k, but beyond that the gains flatten — a batch of 256k or even a million buys little over 32k. The community had assumed ever-larger batches were the road to better CLIP; SigLIP's controlled study shows there is a knee, and it is reachable.
Consider n = 2. The softmax denominator for image 0 is eℓ00 + eℓ01 — a single positive and a single negative. The "contrast" is against exactly one other caption, which may be trivially dissimilar (a corgi vs a rocket), giving almost no learning signal. The sigmoid loss for the same n = 2 still asks the absolute question "is (0,0) a match? is (0,1) a match?" and gets a meaningful gradient from each independently of how many negatives exist. As n grows, the softmax's single-positive-vs-many-negatives contrast sharpens and catches up.
A schematic of the paper's central curve: zero-shot accuracy vs batch size for sigmoid (warm) and softmax (teal). At small batch sigmoid leads; the curves converge and both flatten past a knee near ~32k. Drag to mark a batch size and read off the gap. This is a qualitative reproduction of the trend, not exact numbers.
| Batch regime | Softmax (CLIP) | Sigmoid (SigLIP) | Who wins |
|---|---|---|---|
| Small (< ~4k) | Noisy denominator, few negatives | Well-defined per-pair loss | Sigmoid |
| Moderate (~8k–16k) | Improving | Still ahead, gap narrowing | Sigmoid (slightly) |
| Large (~32k) | Near-saturated | Near-saturated | Comparable |
| Huge (256k–1M) | Saturated — little extra gain | Saturated — little extra gain | Comparable; not worth the cost |
SigLIP is evaluated the standard CLIP way: zero-shot classification. After training only on image-text pairs (no labels), you classify an image by embedding the candidate class names as captions ("a photo of a {class}") and picking the one whose text embedding is closest to the image embedding. ImageNet zero-shot top-1 accuracy is the headline metric.
The paper's most efficient result uses SigLiT — sigmoid loss with a locked (frozen, pre-trained) image encoder and only the text tower trained, following the LiT recipe (Zhai et al., 2022). Because the image features are fixed, training is extremely cheap, and the sigmoid loss's small-batch strength shines: SigLiT reaches strong ImageNet zero-shot accuracy (in the mid-80s percent) after a short training run on only a few accelerator chips. The from-scratch SigLIP (both towers trained) is also competitive with, and at matched compute often better than, softmax CLIP.
Removing the negative bias initialization (starting b near 0) hurts and slows training, exactly as the Chapter 3 imbalance argument predicts — the early gradient is dominated by the negative flood. A strongly negative binit (around −10) is important for stable, fast convergence. The temperature initialization (t around 10) follows CLIP and is similarly important for putting the cosine similarities into a usable logit range.
Holding everything else fixed and swapping only the loss, sigmoid matches or beats softmax across the batch sizes studied, with the largest margin at small batch (Chapter 5). This is the cleanest possible ablation because the only variable is how the logit matrix becomes a loss.
Because each negative is scored independently, the sigmoid loss is more tolerant of the small-batch regime where the softmax has too few negatives to form a sharp contrast. The paper also explores how the loss behaves as data and batch are varied, consistently finding the sigmoid loss a robust default.
| Setting | What is trained | Cost | ImageNet zero-shot |
|---|---|---|---|
| SigLiT | Text tower only (image frozen) | Very low (few chips, ~1–2 days) | Strong (mid-80s %) |
| SigLIP (from scratch) | Both towers | Moderate | Competitive with / better than softmax CLIP at matched compute |
| Softmax CLIP baseline | Both towers | Moderate–high (all-gather) | Strong at large batch, weaker at small |
An image embedding (warm dot) is compared to several class-name text embeddings (teal). The bars are the sigmoid match scores σ(t·cos + b) — each independent. Drag to rotate which class the image is actually of and watch the matched bar rise while the others stay low (no competition, just absolute scores).
This is the payoff. You have a live 6 × 6 image-text similarity matrix. You control the temperature t, the bias b, and how strongly the diagonal (true pairs) separates from the off-diagonal. The panel computes both losses on the same matrix in real time, shows the resulting probability matrices side by side, and reports the two scalar losses. Play until you can predict how each control moves each loss.
Left: the sigmoid probability matrix σ(t·cos + b) — each cell independent, brighter = higher match probability. Right: the softmax (image→text) probability matrix — each row sums to 1, so brightening one cell dims its neighbors. The diagonal is the true pairs (ringed warm). Watch how the bias only shifts the sigmoid side (the softmax is shift-invariant), and how separation sharpens both.
Three experiments to run before you move on:
SigLIP is a loss-function change, not a silver bullet. Several limits and open questions are worth stating plainly.
The clearest win is at small-to-moderate batch. If you already command tens of thousands of accelerators and run batches of 32k or more, the softmax catches up and the practical accuracy difference becomes small. SigLIP does not promise a better large-scale model; it promises an equally good one trained more cheaply and a better one when you cannot afford the giant batch.
Like CLIP, SigLIP learns from noisy web image-text pairs and inherits their biases, mislabelings, and gaps. A different loss does not clean the data. Downstream fairness and robustness issues familiar from CLIP carry over.
The negative bias initialization is important; get it wrong (e.g. binit near 0) and early training is dominated by the negative flood. The loss has a couple of sensitive knobs that the softmax does not, which is a small added burden even though the defaults (b = −10, t = 10) work well.
SigLIP removes the global softmax normalization and its all-gather, but it still scores all n² pairs — the matrix of logits is unchanged. The savings are in communication and memory, not in the asymptotic number of pair evaluations. Very large batches are still quadratic in compute.
| Question | SigLIP's answer |
|---|---|
| Does it beat softmax at huge batch? | No — comparable; both saturate ~32k |
| Does it remove the O(n²) pair compute? | No — same matrix; only the global sync is gone |
| Does it need extra hyperparameters? | Yes — a bias term (init −10), but defaults are robust |
| Does it fix web-data bias? | No — same data, same inherited issues |
| Where is it clearly better? | Small/moderate batch, modest hardware, frozen-image SigLiT |
SigLIP sits at a crossroads of contrastive learning, multimodal pretraining, and distributed-training systems. Here is how it connects to the rest of the map.
SigLIP is the direct successor to CLIP (Radford et al., 2021) and LiT (locked-image tuning, Zhai et al., 2022). It keeps their two-tower architecture and zero-shot evaluation, and changes only the loss. The sigmoid-per-pair idea echoes the prior-bias trick from imbalanced detection (focal-loss-era detectors) and the long line of binary-vs-softmax debates in metric learning.
SigLIP became a workhorse vision encoder: SigLIP and SigLIP 2 image encoders are widely used as the frozen visual backbone in vision-language models (the "image tower" feeding an LLM). Its small-batch efficiency made it a practical default well beyond the original translation-flavored CLIP use case.
| Concept | One-line summary |
|---|---|
| Core change | Replace CLIP's softmax contrastive loss with a per-pair sigmoid (binary) loss on the same logit matrix |
| Logit | ℓij = t · xi · yj + b, with x, y L2-normalized; t, b learnable |
| Label | zij = +1 on the diagonal (true pairs), −1 off-diagonal |
| Loss | L = −1⁄n ∑i,j log σ(zij ℓij) — independent per cell |
| Temperature t | Learnable as t = exp(t'), init t = 10; scales cosine into logit range (the slope) |
| Bias b | Learnable, init −10; counters the n² negative flood at init (the operating point) |
| Why no bias in CLIP | Softmax is shift-invariant — a bias cancels in the normalized row |
| Systems win | No global softmax → no all-gather barrier; chunked ring-pass loss, O(b) memory/device |
| Batch finding | Sigmoid wins at small/moderate batch; both saturate ~32k; bigger batches buy little |
| Best variant | SigLiT (frozen image tower) — strong ImageNet zero-shot on a few chips in ~1–2 days |
| Legacy | SigLIP image encoders are a standard frozen backbone inside modern VLMs |
"What I cannot create, I do not understand." — and you just created both losses, by hand and in code, on the same matrix. You understand the difference.