EVA-CLIP: Improved Training Techniques for CLIP at Scale

Chapter 1: CLIP in One Page

To understand what EVA-CLIP improves, we first need the object it improves. CLIP (Contrastive Language-Image Pretraining) learns a shared embedding space for images and text. The goal: a photo of a dog and the sentence "a photo of a dog" should land at nearly the same point in this space, while the photo and "a photo of a car" should land far apart.

CLIP has two encoders working in parallel:

The crucial step is L2 normalization: both embeddings are scaled to unit length. After normalization, the dot product between an image embedding and a text embedding is exactly their cosine similarity — a number in [-1, 1] that measures alignment regardless of vector magnitude.

Here ‖v‖ means the Euclidean (L2) norm of vector v, and θ is the angle between the two unit vectors. Because both are unit length, their dot product is the cosine of that angle.

Worked example: a tiny 2-D space

Suppose d = 2 (we use 2 dimensions so we can do it by hand). The image encoder outputs the raw vector i = [3, 4], and the text encoder for "a photo of a dog" outputs t = [6, 8].

L2 norm of i: ‖i‖ = √(3² + 4²) = √25 = 5. So î = [3/5, 4/5] = [0.6, 0.8].

L2 norm of t: ‖t‖ = √(6² + 8²) = √100 = 10. So t̂ = [0.6, 0.8].

Cosine similarity: î · t̂ = 0.6×0.6 + 0.8×0.8 = 0.36 + 0.64 = 1.0. Perfect alignment — the image and caption point in exactly the same direction, even though their raw magnitudes (5 vs 10) differed. That is the whole point of normalizing: only direction carries meaning.

At inference, zero-shot classification is just this similarity, repeated. To classify an image into 1000 ImageNet classes, you embed the image once, embed all 1000 captions ("a photo of a {class}") once, and pick the class whose text embedding has the highest cosine similarity with the image. No fine-tuning, no labeled training images for those classes — the alignment learned during pretraining does all the work.

This is exactly why CLIP's quality of representations matters so much, and why EVA-CLIP's bet on a strong pretrained vision backbone is so natural: if the image encoder already produces rich, semantically organized features before contrastive training even begins, you have less work to do to align it with text.

Chapter 2: The Contrastive Loss, Derived

CLIP's training signal is the contrastive loss (a symmetric InfoNCE). The intuition: within a batch of N image-text pairs, the model should make each image most similar to its own caption and less similar to the other N−1 captions in the batch — and vice versa. The other pairs in the batch act as negatives "for free."

Let's build it from the similarity matrix. Take a batch of N pairs. Stack the normalized image embeddings into a matrix Î of shape [N, d] and the normalized text embeddings into T̂ of shape [N, d]. The full pairwise similarity matrix is:

Entry S_ij is the cosine similarity between image i and text j, multiplied by a learned temperature scale. CLIP parameterizes the temperature as exp(τ) where τ is a learnable log-temperature (clamped so the scale does not explode). A larger scale sharpens the softmax; a smaller scale softens it. The diagonal entries S_ii are the correct matches; the off-diagonal entries are the negatives.

The two-way softmax

Now read the matrix two ways. Reading row i: image i should pick text i out of all N texts. This is an N-way classification, and the loss is cross-entropy with the correct class being the diagonal:

Reading column j: text j should pick image j out of all N images:

The final CLIP loss is the symmetric average of the two directions:

This symmetry matters: image-to-text retrieval and text-to-image retrieval are equally important downstream, so both directions are penalized.

Worked example: a 2-pair batch

Take N = 2. Suppose the scaled similarity matrix came out as:

Row 0 (image 0 vs texts 0,1): softmax of [4.0, 1.0]. Subtract the max (4.0): exponents of [0, −3] = [1, 0.0498]. Sum = 1.0498. Probability of the correct text 0 = 1 / 1.0498 = 0.953. So row-0 loss = −log(0.953) = 0.048.

Row 1 (image 1 vs texts 0,1): softmax of [0.5, 3.0]. Subtract max (3.0): exponents of [−2.5, 0] = [0.0821, 1]. Sum = 1.0821. Probability of correct text 1 = 1 / 1.0821 = 0.924. Row-1 loss = −log(0.924) = 0.079.

Image-to-text loss = (0.048 + 0.079) / 2 = 0.064. By the symmetry of this particular matrix you would compute the text-to-image direction over columns and average. The total loss is small here because the diagonal dominates each row — exactly what we want. If the off-diagonal entries were larger than the diagonal, the loss would blow up, pushing the encoders to fix the mismatch.

This is also why batch size matters enormously for CLIP. The negatives come from within the batch: a batch of N gives each image N−1 negatives. Bigger batches mean harder, more diverse negatives, which sharpen the embedding space. OpenCLIP and EVA-CLIP both train with very large batches (tens of thousands), which is precisely why the optimizer and memory techniques in later chapters are so important — you cannot use a 32k batch if the optimizer is unstable or the attention does not fit in memory.

python
import torch
import torch.nn.functional as F

def clip_loss(img_emb, txt_emb, logit_scale):
    """
    img_emb: [N, d]  (raw image embeddings)
    txt_emb: [N, d]  (raw text embeddings)
    logit_scale: scalar = exp(tau), the learned temperature scale
    """
    # 1. L2-normalize so dot product == cosine similarity
    img = F.normalize(img_emb, dim=-1)
    txt = F.normalize(txt_emb, dim=-1)

    # 2. Scaled similarity matrix S [N, N]
    logits = logit_scale * img @ txt.T

    # 3. Correct match for image i is text i -> labels are 0..N-1
    labels = torch.arange(img.size(0), device=img.device)

    # 4. Symmetric cross-entropy over rows and columns
    loss_i2t = F.cross_entropy(logits, labels)        # rows
    loss_t2i = F.cross_entropy(logits.T, labels)      # columns
    return (loss_i2t + loss_t2i) / 2

Chapter 3: EVA — the MIM-Pretrained Backbone

The single most important ingredient in EVA-CLIP is the letters in front of it: EVA. EVA is a Vision Transformer pretrained before any contrastive training, using masked image modeling (MIM). EVA-CLIP then uses this pretrained EVA as the initial image encoder. So to understand the warm start, we must understand what EVA learned.

Masked image modeling is the vision analogue of masked language modeling (the "fill in the blank" objective that trained BERT). You take an image, split it into patches, hide a large fraction of them, and ask the model to reconstruct what was hidden from the visible context. To reconstruct a masked patch, the model must understand objects, textures, and scene structure — so it learns rich visual features as a side effect.

EVA's key trick: reconstruct CLIP features, not pixels

Ordinary MIM (like MAE) reconstructs raw pixels. EVA does something more semantic: its reconstruction target is the visual feature produced by a (frozen) pretrained CLIP image encoder for the masked patch. In other words, EVA is trained to predict "what would CLIP say this hidden patch looks like?"

Why does this matter? Raw pixels are noisy and low-level — predicting them teaches texture, but not much semantics. CLIP features are already semantic (they were aligned to language). By regressing toward CLIP features, EVA inherits CLIP-like semantics and the spatial, dense understanding that comes from MIM. The result is a vision backbone that is both semantically organized and a strong dense feature extractor. This is sometimes summarized as "MIM with a CLIP teacher."

Here EVA_p is EVA's predicted feature for masked patch p and CLIP_p is the frozen teacher's feature for the same patch. The sum is only over masked patches — visible patches are not penalized, because predicting what you can already see is trivial.

There is a beautiful bootstrap here. CLIP supervises EVA (as a teacher). Then EVA initializes the next CLIP (EVA-CLIP). Each generation of vision feature is distilled into a better backbone, which trains a better contrastive model, whose features can teach the next backbone. EVA-CLIP is one turn of that flywheel made explicit and efficient.

Chapter 4: The Warm-Start Initialization

Now the core mechanism of the paper: instead of starting the CLIP image encoder from random weights, EVA-CLIP initializes it from the pretrained EVA backbone. The text encoder is initialized from OpenAI's released CLIP text weights. Then contrastive training proceeds as usual. This is the "warm start."

To see why this is such a large win, trace the data flow at step zero of training.

Aligning vs building

This is the heart of it. A cold-start CLIP must build a good visual representation AND align it to text simultaneously, from noise. A warm-started CLIP already has the representation; it only has to align. Alignment is a far smaller, more stable optimization problem than representation learning from scratch. That is why the loss starts lower and falls faster (the curves you saw in Chapter 0), and why far fewer image-text samples are needed to reach a target accuracy.

What exactly is copied?

The EVA backbone and the CLIP image encoder share the same ViT architecture (patch embedding, the stack of Transformer blocks). EVA-CLIP copies those weights directly. The one part that cannot be copied is the final projection head that maps the ViT's output into the shared d-dimensional contrastive space — that is new and starts random (or lightly initialized), because EVA was trained for reconstruction, not contrast. So the warm start is: copy the heavy feature extractor, learn only the thin projection and the alignment from there.

There is a subtle stability benefit too. A randomly initialized deep ViT produces high-variance activations early on, which interacts badly with large batch sizes and large learning rates — a recipe for loss spikes. A pretrained backbone produces well-conditioned activations from the first step, so the optimizer can use a larger, more aggressive learning rate without diverging. The warm start does not just save samples; it makes the whole training run safer.

Chapter 5: The LAMB Optimizer

The second pillar is the optimizer. EVA-CLIP found that LAMB (Layer-wise Adaptive Moments for Batch training) trains large CLIP models more stably than the usual AdamW, especially at the huge batch sizes CLIP needs. To understand why, we have to look at what LAMB does differently.

Recall AdamW: it keeps a running estimate of the gradient's first moment m (the mean) and second moment v (the variance) per parameter, and updates each parameter by a step proportional to m / (√v + ε). The trouble at large batch sizes is that different layers can have wildly different gradient magnitudes. A single global learning rate that is safe for one layer can be far too large for another, causing instability — exactly the loss spikes that plague cold large-batch training.

The layer-wise trust ratio

LAMB adds one idea on top of Adam: it rescales each layer's update by a trust ratio. For a layer with weights w and Adam-style update direction r = m̂ / (√v̂ + ε), LAMB computes:

Here ‖w‖ is the L2 norm of the layer's current weights, ‖r‖ is the norm of its proposed Adam update, and η is the global learning rate. The effect: each layer's step is scaled so that the update is proportional to the size of that layer's weights. A layer with large weights gets a proportionally larger step; a layer with small weights gets a smaller one. No single layer can take a step that is huge relative to its own scale.

Worked example: two layers, one global LR

Suppose η = 0.01. Layer A has weight norm ‖w‖ = 10 and proposed update norm ‖r‖ = 2. Layer B has ‖w‖ = 0.5 and ‖r‖ = 2 (same raw update, very different weights).

Plain AdamW would move both layers by the same 0.01×2 in update-norm terms — a 0.02 step. For layer B, whose weights are only 0.5 in norm, that is a relative change of 0.02/0.5 = 4%. For layer A it is 0.02/10 = 0.2%. Layer B is being shoved 20× harder relative to itself.

LAMB instead scales by the trust ratio. Layer A: trust = 10/2 = 5, step norm = 0.01×5×2 = 0.1, relative change 0.1/10 = 1%. Layer B: trust = 0.5/2 = 0.25, step norm = 0.01×0.25×2 = 0.005, relative change 0.005/0.5 = 1%. Both layers now change by the same 1% of their own scale. That self-consistency is what keeps large-batch training from blowing up.

Why does this unlock larger batches? Bigger batches give lower-variance gradients, which should let you use a larger learning rate. But a larger global LR amplifies the per-layer mismatch and triggers instability. LAMB removes the mismatch, so you can crank the learning rate (and the batch) without the small-weight layers exploding. That is precisely the regime CLIP lives in: tens of thousands of samples per batch to get hard in-batch negatives.

Chapter 6: Acceleration — Memory & Kernels

The third pillar is engineering. Warm start and LAMB make training need fewer steps and stay stable; acceleration makes each step cheaper in time and memory. EVA-CLIP leans on three tools: flash attention, DeepSpeed ZeRO, and mixed-precision training. None changes the model's math — they change how that math runs on the hardware.

Flash attention: the memory bottleneck of attention

Recall that self-attention computes an N×N attention matrix, where N is the number of tokens/patches. For a high-resolution ViT, N can be hundreds, and the naive implementation materializes that full N×N matrix in slow GPU memory (HBM), reads it back, applies softmax, writes it again. That memory traffic — not the arithmetic — is the real bottleneck.

Flash attention computes the exact same attention output without ever writing the full N×N matrix to HBM. It processes the matrix in tiles that fit in the fast on-chip SRAM, fusing the QK^T, softmax, and weighted-sum steps, and uses an online-softmax trick to keep the running normalization correct as it streams over tiles. The result is identical numbers, much less memory traffic, and a large speedup — which is what lets EVA-CLIP push to larger models and batches on the same hardware.

DeepSpeed ZeRO: sharding the optimizer state

A 5-billion-parameter model does not fit on one GPU once you add optimizer state. Adam/LAMB keep two extra full-size tensors per parameter (the moments m and v). With mixed precision you also keep a high-precision master copy. So the optimizer memory can be several times the model size. ZeRO shards these states (and optionally gradients and parameters) across all the GPUs in the data-parallel group, so each GPU only stores its slice. This is what makes the multi-billion-parameter EVA-CLIP models trainable at all.

The takeaway: acceleration is what turns a good recipe into a feasible run. A method that needs fewer samples (warm start) and trains stably (LAMB) still has to physically fit on GPUs and run fast enough to finish. Flash attention plus ZeRO plus mixed precision are the difference between "trainable on a modest cluster" and "needs a supercomputer."

Chapter 7: The Headline Results

Now the payoff. EVA-CLIP trains a family of models, from a ViT-base-sized version up to the giant EVA-02-CLIP-E/14+ (the "E" denotes an enormous vision tower, "/14" the 14×14 patch size, and the "+" a higher-resolution / longer-schedule variant). The flagship reaches roughly 82% zero-shot top-1 accuracy on ImageNet-1K.

The number itself is strong, but the context is the point. EVA-CLIP reaches this accuracy while seeing far fewer image-text samples and using far fewer GPUs / GPU-days than the comparable OpenCLIP-G model that it matches or surpasses. The paper's central claim is efficiency-at-quality, not quality alone.

The model family scales cleanly

A key empirical finding is that the EVA-CLIP recipe scales: as you grow the model (B → L → g → G → E) and the data, zero-shot accuracy keeps climbing, with no sign of the recipe falling apart. This monotonic scaling is exactly what you want from a training method — it means the techniques are not a one-off trick tuned to a single model size, but a general recipe.

(Tiers and numbers are approximate/illustrative — read them as "the family climbs cleanly to the low-82% range," which is the paper's qualitative result, not exact leaderboard figures.)

Beyond ImageNet

Crucially, the gains are not limited to ImageNet. EVA-CLIP also improves zero-shot performance across a broad suite of classification and retrieval benchmarks, and the EVA-CLIP image encoder transfers well as a frozen backbone for downstream vision-language models. A better contrastive model produces better general-purpose visual features — the same flywheel from Chapter 3, now spinning the other way.

Chapter 8: Ablations — Which Technique Matters?

A bundle of techniques raises an obvious question: which one is actually doing the work? EVA-CLIP's ablations isolate the contributions, and the answer is clear and instructive.

The warm start is the biggest lever

By far the largest single improvement comes from initializing the image encoder from EVA versus random. This is the difference between aligning a good representation and building one from scratch (Chapter 4). Remove it and you lose most of the efficiency gain — you are back to OpenCLIP-style sample budgets.

Optimizer and acceleration are enablers

LAMB and the acceleration stack do not, by themselves, raise the ceiling of accuracy much. What they do is make the large-scale, large-batch regime reachable and stable. Without LAMB, the big models that need huge batches become unstable; without flash attention/ZeRO, they do not fit. So their contribution shows up as "you can now train the model that gets the high accuracy," rather than a direct accuracy bump on a fixed small model.

Other findings worth knowing

• Text-encoder warm start helps too, though less than the image side — initializing from OpenAI CLIP's text weights gives the text tower a head start in the same shared language.
• Freezing vs fine-tuning the backbone. EVA-CLIP fine-tunes the EVA backbone during contrastive training rather than freezing it — the alignment needs to nudge the visual features, not just learn a projection on top of frozen ones. The warm start is a starting point, not a frozen feature bank.
• Scaling robustness. The benefit of the EVA init does not wash out as the model grows; if anything, a strong init matters more at scale, where cold-start instability is worst.

Chapter 9: Training Explorer — Put It All Together

This is the payoff. You now control a simulated CLIP training run with every EVA-CLIP knob exposed. Toggle the warm start, switch the optimizer, change the batch size and learning rate, and watch the loss curve, the running zero-shot accuracy, and a live similarity matrix respond. There is no quiz here — the simulation is the test of whether you understood the previous nine chapters.

Things to try (and what you should see)

• Cold start vs warm start. Switch init to Cold and replay. The loss starts near its maximum and the accuracy crawls for a long time before climbing — you are watching representation-building-from-noise. Switch back to Warm: the loss starts low and accuracy rises immediately. This is Chapter 4 made visible.
• Crank the learning rate with AdamW vs LAMB. Set the optimizer to AdamW, push the LR to the top, and increase the batch — the loss curve develops spikes (instability). Switch to LAMB at the same LR/batch and the spikes smooth out. This is Chapter 5.
• Grow the batch. Larger batches give more in-batch negatives, so the final accuracy ceiling rises slightly — but only if the optimizer is stable enough to use them. This is Chapter 2 meeting Chapter 5.

Chapter 10: Limitations & Connections

EVA-CLIP is a recipe, and like every recipe it has boundaries. Knowing them is part of understanding it.

Limitations

• It depends on a strong teacher. The warm start is only as good as the EVA backbone, which itself was trained against a CLIP teacher. The whole stack inherits whatever biases and blind spots the original CLIP features carry — EVA-CLIP makes training cheaper, not necessarily fairer or more robust to distribution shift.
• It is still a CLIP. The architecture is unchanged, so all of CLIP's known weaknesses persist: poor fine-grained counting, sensitivity to prompt wording, and the bag-of-objects tendency where word order in the caption is underused.
• Compute is reduced, not eliminated. "Far fewer GPUs than OpenCLIP" still means a serious cluster for the largest models. The efficiency is relative.
• The in-batch-negative dependence remains. CLIP still needs very large batches for hard negatives, which is why the acceleration and optimizer work is load-bearing — the method does not remove the appetite for batch size, it just makes feeding it feasible.

The big idea to carry away

EVA-CLIP is the clearest large-scale demonstration of a principle that now pervades deep learning: do not learn from scratch what you can warm-start. A strong self-supervised backbone is a reusable asset. Spend your contrastive compute on alignment, not on rebuilding visual understanding you already have. Combine that with an optimizer that tolerates the batch sizes the loss demands, and kernels that fit the model on real hardware, and you get state-of-the-art at a fraction of the cost.

Where to go next

Related deep dives and lessons on this site:

Cheat sheet