What Makes for Good Views for Contrastive Learning?

Two views of a scene can teach a model the right thing or the wrong thing. Share too much and it learns noise. Share too little and it loses the task. InfoMin finds the sweet spot: 73.0% on ImageNet.

Drag the offset between two image patches. Watch what the patches share, and watch the score rise, peak and fall. Then we build, piece by piece, the idea behind that curve, the experiments that traced it, and the method that learns good views by itself.

You need a feel for what a neural network is (a function with adjustable numbers) and a little probability. We build contrastive learning and mutual information from zero.

A reading of What Makes for Good Views for Contrastive Learning? by Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid and Phillip Isola (NeurIPS 2020, arXiv 2005.10243). It is a reading for Lecture 7 of MIT MMAI, Alignment. Every idea is re-taught here in our own words and devices.

Slide the patches apart

64

The sweet spot at d = 128 is the paper’s (Fig. 3, patches of 64 by 64 from DIV2K images, the second patch starting d pixels right and down from the first). The scene, the bits and the score are a toy model written for this page, shaped like the paper’s reverse-U curve: shared object information falls as the patches drift off the object, shared lighting information falls faster.

Chapter 0

The Views Are the Teacher

See why the pair of pictures you show a model decides what it learns, before any label does

Open the photo album on your phone. It holds forty pictures of one dog. In some the dog sits in sun, in some in shade. It faces left, right, or away. A friend who has never seen the dog still says, “That is the same dog.” Your brain has thrown away the light and the angle and kept the dog.

The writer Jorge Luis Borges imagined a man who cannot do this. In his story Funes the Memorious, Funes finds it unfair that “the dog at three fourteen (seen from the side)” has the same name as “the dog at three fifteen (seen from the front)”. He remembers every view in perfect detail. He cannot collect them into one dog. The paper opens with this story, and it asks a sharp question: how much of a view should a model forget?

Learning without labels, from pairs

Nobody labelled those forty photos. A model still can learn from them. The trick is called contrastive learning, and it is a kind of self-supervised learning (learning from the data itself, with no human labels). Take one scene and make two views of it. A view is any version of the data: a crop, a colour-shifted copy, a patch, a sound clip that goes with a picture. Then train the model so that the two views of one scene land close together, and views of different scenes land far apart.

What the model learns is a representation: a short list of numbers, a code, for each input. A good code puts the same dog near the same dog. Later, someone uses the code for a real job, such as naming the animal. That job is the downstream task. The usual test is a linear readout: freeze the code, and train only one simple layer on top to do the task. If the layer does well, the code already holds what the task needs.

The part nobody had examined

Here is the catch. The model is only ever told, “These two views belong together.” So it keeps whatever the two views have in common, and it drops the rest. You decide what the two views have in common when you build them. If both crops are taken from the same spot, they share the exact lighting, so the model may use the lighting to match them. If the crops come from far apart, they may share nothing useful at all.

The paper by Tian, Sun, Poole, Krishnan, Schmid and Isola studies this choice. Its claim fits in a sentence: a good pair of views shares just the information the task needs, and nothing more. The authors call it the InfoMin principle, for “minimum information”. Too much shared, and the model learns noise. Too little, and it loses the task. You saw this in the instrument at the top of the page: the score rises, peaks, and falls.

The same question sits under the whole lecture on alignment. When a model aligns a photo with its caption, or a video with its sound, each side is a view of the same event. What the two sides share decides what the model can learn. This paper gives one of the clearest answers for images.

Feel it: the best forgetting depends on the job

Try a small experiment before any maths. Below, a model learns from views and slowly becomes invariant to lighting, which means its code stops changing when the lighting changes. At zero, it keeps everything, like Funes. At one hundred, it ignores lighting fully. Pick a job, then slide.

Forget the light, or keep it?

Choose a downstream task. Drag the invariance to lighting. The marker on each bar is where that task does best.

30

Toy curves written for this page, not the paper’s numbers: naming the object gains accuracy as lighting is ignored (70 to 94 percent), telling the time of day loses it (95 down to 25 percent, which is chance for four times of day). The point is the paper’s: the best amount of invariance depends on the task.

For naming the object, more invariance helps, because lighting is a distraction. For telling the time of day, more invariance is a disaster, because lighting is the answer. No single setting is right for both. The paper makes this point twice: with theory, and with a toy dataset of moving digits (Chapter 6).

The road

The paper makes three kinds of claim: a principle, a body of evidence, and a method. We follow them in that order, and we teach the background first.

  1. Learning from pairs: encoders, positives, negatives, the pull and the push.
  2. InfoNCE and mutual information: the loss, and the number it estimates.
  3. The InfoMin principle: sufficient and minimal encoders, three regimes, the reverse U.
  4. Shrinking what views share: crops, colour channels and blur, and the curves they trace.
  5. Augmentation as view design: five famous methods, and the recipe that reached 73.0%.
  6. Views depend on the task: the Colorful-Moving-MNIST experiments and the shortcut.
  7. Learning the views: invertible colour generators, adversarial and semi-supervised.
  8. Results and limits: ImageNet, COCO, PASCAL VOC, and what the paper does not show.
  9. Field Guide: the numbers, a checklist, the core code, and where to go next.
Two crops of the same photo share the exact lighting. Why can that hurt a contrastive model that will later classify objects?

Chapter 1

Learning From Pairs

Pull two views of one image together, push views of other images away, and watch a code form with no labels

Imagine a party game. A host hands you two photo halves and asks, “Do these come from the same picture?” At first you guess. After a hundred rounds you notice cues: the same fur colour, the same shape of ear. You were never told what a dog is. You still learned what a dog looks like, because the game forced you to.

Contrastive learning is that game, played by a network. This chapter builds the game from zero. The next chapters ask which photo halves make the best game.

Encoders turn views into codes

A view is an input such as a 64 by 64 colour patch. As a tensor it has the shape 3 × 64 × 64: three colour channels, 64 rows, 64 columns. An encoder is a neural network that maps a view to a short vector. For a batch of B views, the input is B × 3 × 64 × 64 and the output is B × D. D might be 128. Each row is the code (the paper calls it the representation) of one view.

The paper writes two views of the same data as v1 and v2. Two encoders turn them into z1 = f1(v1) and z2 = f2(v2). The two encoders may share weights when both views come from the same kind of data, such as two crops. They may not share weights when the views differ in kind, such as a picture and a sound clip.

Scoring a pair

We need one number that says how well two codes match. A common choice is the cosine similarity: the cosine of the angle between the two vectors. It is 1 when they point the same way, 0 when they are at right angles, and −1 when they point opposite ways. Dividing by a small number τ (the temperature) sharpens the score. The paper calls this score function h.

h(v1, v2)
The score of a pair of views. A high score means “these two belong together”.
cos(z1, z2)
The cosine of the angle between the two codes, from −1 to 1.
τ
The temperature. A small τ makes the score more sensitive to small differences in angle.

The paper leaves h general: it only says the score typically consists of the two encoders. The cosine form is the common practice in the methods it studies. We use it here because it is easy to see.

Positives, negatives and the question the model answers

Take a batch of N images. Make two views of each. For image i, the view pair (v1,i, v2,i) is the positive pair. The views of every other image j are negatives, also called distractors. For each v1,i the model gets a multiple-choice question: “Which of these N second views came from my image?” It scores all N, turns the scores into probabilities with a softmax, and is rewarded for putting high probability on the right one.

Work one example with three choices and τ = 0.5. The true partner has cosine 0.9. The two negatives have 0.2 and 0.1. The scores are 1.8, 0.4 and 0.2. Their exponentials are 6.05, 1.49 and 1.22, which sum to 8.76. The partner gets 6.05 / 8.76 = 69.0% of the probability. The loss is the negative log of that share: −ln 0.690 = 0.37.

Lower the temperature to τ = 0.1. The scores become 9, 2 and 1. The exponentials are 8103, 7.39 and 2.72. The partner now takes 99.87% and the loss is 0.0013. A smaller τ makes the same angles look much more decisive. That is why τ is one of the knobs these papers tune (the appendix uses 0.07 for a linear critic head and 0.15 for a non-linear one).

Now try it. Four images, each with a view 1 (a filled dot) and a view 2 (a hollow square), placed on a circle by an angle. A training step nudges all eight angles downhill on the loss. The right panel shows the share each partner gets.

Pull and push

Dots are view 1, squares are view 2, letters A to D are the four images. The lines join true partners. Press Train 30 steps and watch partners meet while the letters spread apart.

0.50

Toy: four images, one angle per view, cosine similarity of angles, plain gradient descent with the gradient found numerically. A random guess among four choices costs ln 4 = 1.39. The batch of four stands in for the paper’s batches of many negatives (the appendix uses 65,536 negatives for MoCo-style training).

When the game breaks

Suppose the encoder outputs the same code for every view. Then every cosine is 1, every score is equal, and every candidate gets 1/N of the probability. The loss sits at ln N, the guessing level. Symptom: the loss does not fall below ln N. Measurement: the similarity grid is flat. Fix: the negatives. Because other images must score lower than the partner, a constant code cannot win. This is why contrastive learning needs the push as well as the pull.

There is a second, quieter failure, and it is this paper’s subject. The model is free to use any cue that tells the partner from the negatives. If both views carry the same colour cast, a colour cast is enough. The loss is happy. The code is poor. The next chapter gives a number to this idea of “what the two views share”.

A contrastive model outputs the same code for every input. What loss does it reach with N candidates, and why?

Chapter 2

InfoNCE and Mutual Information

Count the bits two views share, and see why the contrastive loss measures exactly that, up to a ceiling

You and a friend each hold a card from a shuffled deck. You ask, “Is your card red?” She says yes. Now you know a little about her card. If she says, “It is the queen of hearts,” you know everything. The amount your card tells you about hers is what this chapter names and measures.

Bits, entropy and mutual information

Start with a bit: the answer to one fair yes-or-no question. Picking one of 8 equally likely options takes 3 questions, so it holds 3 bits. The entropy H(X) of a variable is its average number of bits of surprise. For 8 equal options, H = log2 8 = 3 bits.

Now give someone a second variable Y that depends on X. After they see Y, some surprise about X remains. That is the conditional entropy H(X | Y). The bits that Y removed are the mutual information.

I(X; Y)
The mutual information, in bits: how much seeing one variable tells you about the other. It is 0 when they are independent.
H(X)
The bits of surprise in X before you see Y.
H(X | Y)
The bits of surprise left in X after you see Y.

For the paper, X and Y are the two views, v1 and v2. So I(v1; v2) is the number of bits the two views share. This is the quantity the whole paper turns up and down. It counts everything shared, useful or not: the object, the lighting, the camera noise.

A worked example you can check

Let x be one of 8 classes, all equally likely (3 bits). View 1 is x itself. View 2 is x with probability 0.8, and otherwise a random class (which may happen to equal x). Then P(v2 = x) = 0.8 + 0.2/8 = 0.825, and each of the 7 other classes has probability 0.2/8 = 0.025.

The leftover entropy is H(v2 | v1) = −0.825 log2 0.825 − 7 × 0.025 log2 0.025 = 0.229 + 0.931 = 1.160 bits. The marginal entropy of v2 is 3 bits. So the views share I = 3 − 1.160 = 1.84 bits. Add more noise and the number falls toward 0. Remove the noise and it climbs to 3.

The InfoNCE loss counts those bits

We cannot compute I for real images. Nobody knows the true joint distribution of two crops. The contrastive loss gives a way around: a lower bound, a number guaranteed not to exceed the truth. The loss is called InfoNCE (NCE is noise-contrastive estimation). For an anchor v1,i, the model must pick its true partner v2,i out of K candidates.

LNCE
The loss: the average negative log of the probability the model gives the true partner.
eh(v1,i, v2,i)
The exponentiated score of the true pair, the one we want to be large.
∑j eh(v1,i, v2,j)
The same for all K candidates, true partner included. The ratio is the softmax share from Chapter 1.

The paper states the link in one line: I(v1; v2) ≥ log K − LNCE = INCE(v1; v2). This is a known result from the original InfoNCE work. Two edge cases show why it is sensible. If the views are independent, the model can only guess, the loss is log K, and the bound reads 0 bits. If v2 pins down v1 perfectly, the loss goes to 0, and the bound reads log K.

The second case shows a limit. The bound can never exceed log K. With K = 8 candidates the ceiling is 3 bits; with K = 2 it is 1 bit. Try it: the views share 1.84 bits at 20% noise. Slide the noise and the number of candidates, and compare the bound to the truth.

Count the bits

Bars: the true bits shared, the InfoNCE estimate, and the ceiling log2 K. Move the noise, then the number of candidates K.

20%

Toy, computed live: eight equally likely classes, view 1 is the class, view 2 is the class or (with the noise probability) a random class. The true bits are exact. The InfoNCE estimate uses the ideal critic h = log(P(v2 | v1) / P(v2)) on 4,000 fixed random trials, so it wobbles by a few hundredths of a bit.

At K = 2 the estimate stops at 1 bit however clean the views are. At K = 64 it can follow the truth all the way to 3 bits. The estimate also never sits above the truth, which is what “lower bound” promises, up to the small Monte Carlo wobble.

How the paper uses it

The paper cannot compute the true I for images, so it uses INCE from a neural critic as a stand-in. It warns that this number depends on the network architecture. So in each plot the authors change only the input views and keep every other setting fixed, which makes the plots comparable. When you read “INCE decreases” in later figures, read “the views share fewer bits, as the critic can tell”.

Failure mode. Symptom: INCE stays flat at one value although the views are clearly more alike. Measurement: compare it with log K for your K. If they match, you hit the ceiling. Fix: use more negatives, or compare views only at the same K.

You train with K = 16 candidates and the InfoNCE estimate sits at 4.0 bits for two very different view recipes. What is the likeliest explanation?

Chapter 3

The InfoMin Principle

Learn why the best views share exactly the task’s bits, and why the score traces a reverse U

You are about to take an exam on one subject, and you may bring one page of notes. A page that leaves out a topic costs you marks. A page that holds every lecture word for word is too long to use, and the key facts drown. The best page holds exactly what the exam asks, no more and no less.

A code is a page of notes, and the two views decide which facts can make it onto the page. This chapter turns that picture into three definitions, one proposition and a curve.

Three definitions, in plain words

The paper first defines which encoders are good for the contrastive game. An encoder f1 is sufficient if the code z1 = f1(v1) tells you as much about v2 as the full view v1 did: I(v1; v2) = I(f1(v1); v2). Nothing the game needs was lost in the squeeze. A sufficient encoder is minimal if, among all sufficient encoders, it keeps the least information about its own view: I(f1(v1); v1) is as small as it can be. It keeps what the game needs and nothing else.

The third definition is about the real job. For a task with label y, the optimal representation z* is the minimal sufficient statistic of y: it holds everything about y that x held, and nothing more. A classifier built on z* can do as well as one built on the raw input, and it has less irrelevant detail to overfit. The paper cites the work of Soatto and Chiuso for why this helps generalization.

Now connect the two. The contrastive game only ever teaches the encoder to keep what the two views share. So if the encoders are minimal and sufficient, the code holds exactly the shared information, I(v1; v2). Whether that is good for the task depends on whether the shared bits are the task bits.

Three regimes

Let I(x; y) be the bits of task information that the input holds. Compare it with I(v1; v2), the bits the views share. The paper names three cases, from the information bottleneck literature.

  1. Missing information. I(v1; v2) < I(x; y). The views share less than the task needs, so some task bits are gone. Performance drops.
  2. Sweet spot. I(v1; y) = I(v2; y) = I(v1; v2) = I(x; y). The views share the task bits and nothing else.
  3. Excess noise. I(v1; v2) > I(x; y). The views also share nuisance bits. The code keeps them, and generalization gets worse.

Plot transfer performance against I(v1; v2) and the three regimes form a hill: up through missing information, a peak at the sweet spot, then a slide down through excess noise. The paper calls it a reverse U and says it upper-bounds what views can reach.

The proposition

Suppose we knew the task. Then the best views would solve a small optimization problem.

(v1*, v2*)
The optimal pair of views, made from the data x.
I(v1; v2)
What we minimize: the bits the two views share.
I(v1; y) = I(v2; y) = I(x; y)
What we hold fixed: each view keeps all the task bits that the input had.

In words: keep every task bit in both views, then make the views as different as you can otherwise. The paper’s Proposition 3.1 says that with minimal sufficient encoders, the learned code is then optimal for the task. The appendix proof shows that at this optimum the two views share exactly I(x; y) bits, and are independent of each other once you know the label. All they share is the label.

Here is the catch the paper admits. Contrastive learning has no labels at training time, so you cannot check the constraint. In practice, people build views by domain knowledge: change the input in ways that should not change the task. The rest of the paper is about doing this well.

Play with the curve. The task holds 4 bits in this toy. Slide the shared bits, and switch between two families of views.

Walk the reverse U

The dot is your pair of views. Left of the green line the views lose task bits. Right of it they carry extra noise. Choose how the views were built, then slide.

6.0

Toy, shaped like the paper’s Fig. 1b and Fig. 2: the task holds I(x; y) = 4 bits (an illustrative number). When the views keep the task bits, the score climbs to 95 at 4 bits and then slides 3 points per extra bit. When they lose task bits, extra sharing never repairs the loss and the peak stays near 70.

With views that keep the task bits, the best score is at 4 bits, the task’s own information. Cutting sharing further hurts; keeping more hurts too. With views that lose task bits, you can make the shared bits small, but the curve never reaches the peak. This is the paper’s warning: lower mutual information helps only while the task bits survive. The appendix shows both shapes on a real experiment (Chapter 4).

How this differs from InfoMax

An older principle, InfoMax, says a code should capture as much information about its input as it can. InfoMin does not reject that. It adds a condition: information is worth capturing only if the task needs it. Past that point the paper argues throwing information away is better, because it improves generalization and lowers the number of examples a downstream task needs.

Two views share 7 bits. The task needs 4 bits, and both views hold all 4. According to the three regimes, what is the situation, and what is the remedy?

Chapter 4

Shrinking What Views Share

Follow four experiments that turn one knob and trace how the score responds as the shared bits fall

A principle is only worth keeping if you can test it. The authors need a way to move I(v1; v2) up and down while changing little else, then watch the downstream score. This chapter walks through the knobs they used. Each knob is a different way of making two views less alike.

The method, the same each time

Every experiment follows one recipe. Build two views with a knob set to some value. Train an encoder with the contrastive loss, with a small network and the same settings each time. Record INCE at the end of training as a stand-in for the bits shared (Chapter 2). Then freeze the encoder, train a linear classifier on top, and record accuracy. Repeat for many knob values. Plot accuracy against INCE. A reverse U in that plot is the InfoMin signature.

Experiment 1: how far apart are the patches?

Take a high-resolution image and cut two patches of 64 by 64 pixels. The first starts at a random point (x, y). The second starts at (x + d, y + d). When d is small, the patches overlap, and they share colour, texture, lighting and often the same object. When d is large, they sit far apart, and little carries over. The authors sweep d from 64 to 384 on images from DIV2K, a high-resolution dataset. They chose it because small images would push big-d patches against the border and bias the result.

Encoders are a tiny AlexNet trained for 3000 epochs. The test is a linear classifier on STL-10 and on CIFAR-10 (resized to 64 by 64). The result in Fig. 3 is a clear reverse U on both datasets, with the sweet spot at d = 128. That is the curve in the page’s opening instrument.

Experiment 2: which colour channels does each view see?

A colour space is a way to write each pixel as three numbers. In RGB the numbers are red, green and blue, and they are strongly correlated: a bright pixel is bright in all three. In YDbDr and Lab the first channel is brightness (luminance) and the other two are colour (chrominance), and these are much less correlated. Split the channels into two views: view 1 is one channel, view 2 is the other two. The less the channels correlate, the fewer bits the views share.

The authors test {R | GB}, {Y | DbDr} and others, on STL-10 classification and on NYU-Depth-v2 segmentation (frozen encoder, a four-layer decoder on top). In Fig. 4, accuracy keeps rising as INCE falls. There is no downturn. That looks like it contradicts the reverse U. It does not: it only says hand-made colour spaces never push far enough to reach the right side of the hill. Chapter 7 builds a way to push further, and then the full U appears.

Experiment 3: low and high frequencies

Blur an image with a Gaussian of width σ. The blurred copy is the low-frequency view: broad shapes. Subtract it from the original and you get the high-frequency view: edges and fine texture. At tiny σ the blur does nothing and the high-frequency view is empty. At huge σ the blur wipes everything and the low-frequency view is a blob. The shared bits peak in between. The paper finds the maximum of INCE at σ = 0.7 on STL-10.

From that peak you can walk in two directions, and they behave differently (Fig. 9). Raise σ above 0.7, and INCE falls while accuracy first rises and then drops: a reverse U with its sweet spot at σ = 1.0. Lower σ below 0.7, and INCE also falls, but accuracy only drops. Both walks reduce the shared bits. They differ in which bits go. The appendix draws the lesson: mutual information is not the whole story. What information is shared also matters. This matches the two shapes in the last chapter.

Experiment 4: the strength of one augmentation

On ImageNet the authors vary one augmentation at a time, using the MoCo framework (65,536 negatives, 100 epochs, batch 256). Two results stand out (Fig. 5, and appendix Figs. 13 and 14).

Try each knob. The shapes below are toys drawn to match the sweet spots reported in the paper; the paper’s plots give the shape, not the table of values.

Turn the knob, trace the curve

Top: toy score. Bottom: toy shared bits. The green line is the paper’s sweet spot for this knob. Pick an experiment, then slide.

64

Sweet spots are the paper’s: d = 128 (Fig. 3), σ = 1.0 with the shared-bits peak at 0.7 (Fig. 9), c = 0.2 and x = 1.0 (Figs. 5, 13, 14). The curves and numbers on the axes are a toy written for this page. For σ below 0.7 the score only falls, as in the paper’s Fig. 9.

Failure mode. Symptom: accuracy keeps rising as you shrink the shared bits, with no peak, as in the colour-space plots. Measurement: you have not yet crossed the sweet spot, or your knob cannot reach it. Fix: use a stronger knob, or a learned one (Chapter 7). Do not conclude that less is always more.

In the blur experiment, lowering and raising σ from 0.7 both reduce INCE, yet only one direction traces a reverse U. What does that show?

Chapter 5

Augmentation as View Design

See five famous methods as five ways to lower shared bits, then climb the ladder that ends at 73.0%

Photocopy a page. Then photocopy the copy. The second copy is a little blurrier, and no copy can have more detail than the original. The same fact holds for views. When you take a view and process it further, the new view cannot tell you more about anything than the one you started from.

This simple fact lets the paper explain a whole generation of methods in one sentence each.

Why processing can only lower shared bits

Information theory has a name for the photocopy rule: the data processing inequality. For any function g, I(g(a); b) ≤ I(a; b). Applying a function to a variable, or adding randomness to it, cannot create information. So if two views share I bits, then a new view made by shuffling, dropping channels, or blurring one of them shares no more than I.

The paper uses this to compare methods. Every recent contrastive method makes two views v1 and v2 of an image, and they all differ in how they lower the shared bits. The common recipe is three steps: sample an image X; sample two transformations t1 and t2 from a family T; and set v1 = t1(X) and v2 = t2(X).

MethodHow it makes its two viewsWhy the shared bits are lower
InstDis, MoCoApply a random augmentation twice to the same image.The baseline: two independent random augmentations.
CMCView 1 is the first colour channel. View 2 is the last two channels.Each view is a function of an InstDis view, so I(v1 cmc; v2 cmc) ≤ I(v1; v2) is guaranteed.
PIRLKeep view 1. Shuffle view 2 like a jigsaw puzzle.The shuffle h adds randomness, so I(v1; h(v2)) ≤ I(v1; v2).
SimCLRUse a stronger family of augmentations T′.Stronger augmentation leaves less in common than InstDis.
CPCTake local patches, with strong augmentation such as RandAugment.Disjoint patches share less, as in the patch-offset experiment.

The paper reports that CMC does better than InstDis and that SimCLR’s stronger augmentations do better again. This table is the paper’s argument, with a caveat in its own words: these methods differ in architecture, objective and engineering tricks too, so the comparison is suggestive rather than a clean test.

Building a stronger recipe: InfoMin Aug

If lower shared bits help, push further. The authors collect the knobs of Chapter 4 into a recipe they call InfoMin Aug: RandomResizedCrop, Color Jittering, Gaussian Blur, RandAugment (a random mix of image edits), Color Dropping (turning the image grey), and a JigSaw branch as in PIRL. Fig. 12 shows that as they strengthen the family step by step, INCE falls and accuracy climbs, and the best of them (called RA-CJ-Blur) still has not reached the sweet spot.

The appendix gives the ladder, one rung at a time, on ResNet-50 with a linear readout on ImageNet:

  1. RA-CJ-Blur, 100 epochs: 63.6%.
  2. Add the JigSaw branch: 65.9%.
  3. Replace the linear projection head with a two-layer MLP: 67.3%. (With this head, a larger temperature helps.)
  4. Train for 200 epochs: 70.1%. That beats SimCLR’s 69.3% at 1000 epochs.
  5. Train for 800 epochs: 73.0%, the new state of the art in the paper. Top-5 reads 91.1%.

Each ingredient removes a different kind of shared detail. A crop with a low area bound c keeps only a small piece, so the two views often show different parts of the object. Colour jitter shifts brightness and hue, so the two views disagree about the colour cast. Blur removes fine texture from one view. Colour dropping turns the image grey, so colour cannot be the match cue. JigSaw shuffles tiles, so the exact layout cannot be the cue. Each one closes a shortcut of the kind Chapter 6 showed.

A cost note from the paper: SimCLR needed 128 TPUs for its large batches. The InfoMin model trains on as few as 4 GPUs in one machine. Step through the ladder and compare with the other methods in Table 1.

Climb the ladder

Bars: ImageNet top-1 linear readout, ResNet-50. The bottom bar is InfoMin at the rung you choose. The line is SimCLR.

All numbers are the paper’s (Table 1 and the appendix text). The first three rungs are 100-epoch runs. InstDis 56.5, Local Aggregation 58.8, CMC 60.0, MoCo 60.6, PIRL 63.6, CPC v2 63.8, SimCLR 69.3 (1000 epochs). Epoch counts for each baseline are in the paper’s Table 1; the text states SimCLR’s as 1000.

Failure mode. Symptom: you strengthen an augmentation and accuracy drops. Measurement: the pair now shares less than the task needs, so you have crossed from the right side of the hill to the left. Fix: back off to the value where the score peaked. The crop bound shows this: 0.2 beats the supervised default of 0.08 by more than a point, and pushing below 0.2 stops helping.

PIRL keeps view 1 and shuffles the patches of view 2. Why does the paper say the shuffle cannot increase the bits the views share?

Chapter 6

Good Views Depend on the Task

Run the paper’s toy world of moving digits, and see which shared factor the model learns, and which it ignores

A friend describes a film to you in a hurry. If you ask her about the plot, she tells the plot. If you ask her about the music, she tells the music. Whatever the two of you talked about last is what she remembers best. A contrastive model is that friend. Whatever the two views share is what it keeps.

Chapter 3 claimed that the best views depend on the task. Natural images make this hard to see, because many factors change at once. So the authors built a small world where each factor can be switched on and off.

The Colorful-Moving-MNIST world

Take Moving-MNIST: a handwritten digit slides across a black 64 by 64 canvas at constant speed and bounces off the walls. The authors replace the black canvas with a fixed natural image, a random 96 by 96 crop from STL-10. Now every frame has three factors that vary independently:

The digit is 28 by 28 pixels. Its start point is random, its direction is random, and its speed is 0.1 of the canvas size per step.

How the pair is built

View 1 is the first 10 frames of a clip, x1:10: a short video. View 2 is a single image built to match frame x20. The authors control the shared bits exactly. For each pair, they copy some factors from x20 into view 2 and fill the others with random values. If only the digit is copied, the views share the digit class and nothing else. If digit and background are copied, they share both.

The data flow, with B as the batch size (the paper uses B = 128):

  1. View 1: B × 10 × 3 × 64 × 64. A four-layer ConvNet with 8, 16, 32 and 64 filters turns each frame into a 64-number vector, so we get B × 10 × 64.
  2. A one-layer LSTM (a network that reads a sequence step by step) with 64 hidden units reads those 10 vectors and outputs B × 64.
  3. View 2: B × 3 × 64 × 64 passes through a ConvNet and becomes B × 64.
  4. Score all B × B pairs. The diagonal holds the true pairs; the other 127 in each row are negatives (an “intra-batch” contrast). Train 200 epochs at learning rate 0.03.

After training, the authors freeze the encoder and train a linear head for each of three tests: classify the digit, classify the background, and locate the digit. The first two report an error rate (chance is 90%, since there are 10 classes). The third reports the average distance error in pixels. A supervised network trained end to end reaches 3.4%, 45.3% and 0.93 pixels.

One factor shared: you learn what you share

In Table 4, when the views share only the digit, digit error is 16.8%, but background error is 88.6%, almost chance. Share only the background and the background error drops to 51.7%, but digit error is 88.6%. Share only the position and the localization error is 3.95 pixels, the best of any contrastive row, while the digit is only partly recovered (57.9% error). The lesson is plain: the shared factor is the one that gets learned. A pair that is perfect for one task is almost useless for another.

Several factors shared: one can swallow the rest

Now share two or three. The result is less friendly. Whenever the background is in the shared set, digit error is about 88% and localization error is about 16 pixels, as bad as sharing the background alone. The background wins even when the digit and position are also shared. When the views share only digit and position, digit error is 14.5% and localization error is 13.7 pixels: the digit is learned, and the position is left behind.

The authors suggest a reason, with a hedge (“this might be because”). The background is a whole natural picture, which holds far more bits than a digit class (a choice of one in ten is only about 3.3 bits). The encoder takes the easiest way to match a pair, which is to match the factor with the most bits to spare. It uses the background as a shortcut. Try every combination below.

Choose what the views share

Switch factors on and off. The bars are the error of a linear probe on each task, lower is better. The green tick is the supervised baseline.

All numbers are the paper’s Table 4 (Colorful-Moving-MNIST, frozen backbone, linear head), read from the extracted table: error rates for digit and background classification (chance is 90%) and error in pixels for localisation. The supervised row is 3.4%, 45.3% and 0.93 pixels.

Failure mode. Symptom: the digit probe sits near 88% error although the digit is in the shared set. Measurement: the background is also shared, and the probe for background is far better than chance. Fix: remove the dominant factor from the shared set, for example by giving each view a different random background, so the shortcut is closed and the digit becomes the best remaining match cue.

Views share digit, position and background. Digit classification error is about 88%. What explains it?

Chapter 7

Learning the Views

Train a small network that rewrites an image’s colours so its two halves share the right bits

A tailor cuts a coat from a roll of cloth. A bad tailor cuts the same way for every customer. A good tailor measures first. Hand-designed augmentations are the one-cut-for-everyone tailor. They work well on ImageNet because many people tuned them over years. On a new kind of data, no one has tuned anything yet.

The paper offers a tailor that measures. It learns the views, using the principle of Chapter 3 as the objective. The experiments stay small and clear: colour channels, where we already know how to measure progress.

An invertible colour generator

Chapter 4 split an image into channels: say luminance as view 1 and the two chrominance channels as view 2. Instead of picking a colour space by hand, learn one. Let g be a neural network that maps each pixel’s three numbers to three new numbers, and call the result a new “colour space”. Split the output into two views: v1 = g(X)1, one channel, and v2 = g(X)2:3, two channels.

The shapes: the input is B × 3 × H × W. The generator keeps that shape. View 1 is B × 1 × H × W and view 2 is B × 2 × H × W. Two separate encoders f1 and f2 read them. This is why the paper edited the first layer of a tiny AlexNet to take 1 or 2 channels.

The generator g is built from flow blocks (invertible networks from the NICE, RealNVP and Glow line of work). Each block uses only 1 by 1 convolutions and ReLU activations, so it acts on each pixel on its own, like a colour conversion. The paper tries a volume-preserving (VP) and a non-volume-preserving (NVP) version, and alternates which channel plays the role of the first one as blocks stack. The key property is that the map is invertible: nothing is lost, only rearranged. This matters for a reason we meet next.

Unsupervised: make the views share fewer bits

The goal is to lower the bits the views share. We can measure that with INCE. So we set up a game with two players, like a GAN (generative adversarial network). The critic encoders f1 and f2 try to raise INCE: they get as good as they can at telling true pairs apart. The generator g tries to lower it.

ming maxf
Two players. The encoders f maximise the bound. The generator g minimises it. Each step takes one gradient move for each.
INCE
The bound from Chapter 2, computed on the generated views. It stands in for the bits shared.

Why invertible? If g could throw information away, the easiest way to make the views share nothing would be to output a constant. The game would end with empty views. An invertible g cannot erase anything, so the generator is forced to find the colour recipe that spreads the information across channels without hiding it.

Run on STL-10 (100,000 unlabeled and 5,000 labeled images), the result matches the theory. INCE and accuracy trace a reverse U (Fig. 7a). Starting from YDbDr the generator stays near the sweet spot. The authors note this agrees with the human prior that luminance plus chrominance is a good split, and speculate that human colour perception may be near-optimal for self-supervised learning. They present that as a conjecture.

But the plain game has two faults. In most runs it lowers INCE too far, past the sweet spot. And like a GAN, it is unstable: runs with identical settings differ a lot. The authors give a cause: the generator has no idea which bits are the task bits, so the constraint I(v1; y) = I(v2; y) = I(x; y) from Proposition 3.1 is badly broken.

Semi-supervised: keep the label bits

The fix uses a few labels, only to guard the task bits. Add a classifier on each view and penalise it for failing to predict the label y.

ming, c maxf
The generator and the two label classifiers c1, c2 minimise; the contrastive encoders maximise.
INCE
Unsupervised term, run on every image: lower the shared bits.
Lce(c1(g(X)1), y)
Supervised term, run on labeled images only: view 1 alone must still predict the label. This keeps I(v1; y) high.
Lce(c2(g(X)2:3), y)
The same for view 2, keeping I(v2; y) high.

Each training iteration draws an unlabeled batch and a labeled batch. When training ends, the authors freeze g, make views with it, and run the ordinary contrastive learning from Chapter 1 followed by a linear readout. The labels are used only to shape the views, not to train the final encoder. Learned views now centre on the sweet spot for any starting colour space and either flow type (Fig. 7b). Try the idea in a one-knob toy.

Train the view generator

One knob a says how far the generator decorrelates the channels. Pick the objective, press Run 60 steps, and watch the dot slide downhill.

Toy, written for this page: shared bits I(a) = 0.3 + 3.2(1 − a)1.6; the label term is 40 times the square of how far a is past 0.5; the score peaks at a = 0.5. Plain gradient descent from a = 0.1. The two real objectives are the paper’s Eqs. 2 and 3.

Without the label term, the only force is to lower the shared bits, so the dot rolls to the far end, where the score collapses. With it, the label penalty rises steeply past the sweet spot and holds the dot there.

What the numbers say

Table 5 compares view generators on STL-10 classification (mean ± spread over runs). Values below are read from the table as extracted:

View generator (images used)RGB inputYDbDr input
Raw views (none)81.5 ± 0.286.6 ± 0.2
Unsupervised (100k)82.4 ± 3.284.3 ± 0.5
Supervised only (5k)79.9 ± 1.578.5 ± 2.3
Semi-supervised (105k)86.0 ± 0.687.0 ± 0.3

Read it in three steps. The unsupervised game barely helps and is noisy (the 3.2 spread). Labels alone (the supervised generator) do worse than raw views, so labels do not replace the contrastive term. Both together win: the paper says the semi-supervised generator significantly outperforms the supervised one, which validates the unsupervised term. With a larger backbone (Fig. 8), the learned views g(RGB) beat raw RGB by a wide margin and reach 94% accuracy.

Failure mode. Symptom: two runs of the unsupervised game with the same settings differ by several points. Measurement: the spread across runs, 3.2 for RGB in Table 5 against 0.6 with labels. Fix: the label term. The appendix adds that with labels the training is stable across learning-rate pairs, which the authors call an advantage.

Why does the unsupervised min-max game often lower INCE too far, and what does the label term change?

Chapter 8

Results, Ablations and Limits

Read each table and figure for what it proves, then list what the paper does not show

A good reader asks of every result: compared with what, measured how, and what else could explain it? This chapter takes the paper’s evidence in turn and gives each piece a one-line verdict.

What each table and figure shows

EvidenceWhat it showsWhat it does not show
Fig. 3, patch offsetA reverse U on STL-10 and CIFAR-10, sweet spot at d = 128.That the same d is best on other data.
Fig. 4, colour channelsAccuracy keeps rising as INCE falls, for classification and segmentation.A downturn. Hand-made splits never reach the right edge of the hill.
Fig. 5 and appendix Figs. 13, 14Sweet spots for crop (c = 0.2) and jitter (x = 1.0) on ImageNet.That these hold for other networks or tasks.
Fig. 9, blurTwo walks from σ = 0.7: one traces a reverse U, one only falls.A rule for choosing which bits to remove.
Table 173.0% top-1 on ImageNet linear readout, ResNet-50.That the gain comes from lower shared bits alone. Epochs and heads changed too.
Tables 2 and 3Transfer to detection and segmentation beats supervised pre-training.Large gains on COCO: they are about one point.
Table 4Shared factors set what is learned, and one factor can dominate.That natural images behave the same way.
Table 5, Figs. 7 and 8Learned colour views beat raw views, and need the label term to be stable.Learned views for ImageNet or for other kinds of views.

Does the representation transfer?

Linear readout on ImageNet is one test. Another asks whether the code is a good place to start fine-tuning for a harder job. The authors fine-tune the pre-trained network for object detection and instance segmentation on COCO (Mask R-CNN, ResNet-50 with a feature pyramid network, with synchronized batch norm added), and for object detection on PASCAL VOC (Faster R-CNN, R50-C4, 24,000 iterations, 16 images each, averaged over 5 runs). All the unsupervised methods in the comparison are pre-trained for 200 epochs.

In COCO with the 1x schedule, InfoMin reaches 40.6 box AP against 39.7 for supervised pre-training, a gain of 0.9, and 36.7 mask AP against 35.9. MoCo v2 sits between them at 40.1 and 36.3. On the 2x schedule, InfoMin leads by 0.9 box AP (42.5 against 41.6). On VOC, InfoMin gains 1.4 AP50, 4.1 AP and 5.8 AP75 over supervised. The appendix adds that a ResNeXt-152 pre-trained with InfoMin on ImageNet-1k beats its supervised twin pre-trained on ImageNet-5k, a dataset six times larger. Check the bars.

Compare pre-training methods

Pick a metric. Teal marks supervised ImageNet pre-training, the baseline every row is compared with.

All numbers are the paper’s Tables 2 and 3. The bar axis starts below the smallest bar of each metric, so small gaps look large; read the printed numbers. Random initialisation is not drawn; it scores 32.8, 29.9, 38.4, 34.7, 60.2, 33.8 and 33.1 on the seven metrics in order.

The ablations in one place

The paper’s smaller experiments each remove one piece. Drop the label term from the view generator and STL-10 accuracy with RGB input falls from 86.0 to 82.4, with a wide spread across runs. Drop the unsupervised term instead and it falls to 79.9, below raw views at 81.5. Use c = 0.08 instead of 0.2 for the crop, and with a non-linear head the accuracy loses more than 1 point. Swap the linear head for a two-layer MLP and ImageNet accuracy rises from 65.9 to 67.3. Add JigSaw and it rises from 63.6 to 65.9. Each result supports the same story: the right amount of sharing matters, and the details of the recipe move the score by a point or two.

What the paper does not show

The last point is a design warning, not a footnote. Every time you pick a view, you pick what your model is able to learn and what it is blind to.

Which statement about the 73.0% ImageNet result is best supported by the paper?

Chapter 9

Field Guide

Keep the numbers, the checklist and the core code, and see where the paper sits in the field

The paper in one paragraph. Contrastive learning keeps whatever two views share, so the views are the teacher. Mutual information, estimated by InfoNCE, counts the shared bits. Performance follows a reverse U: too few shared bits lose the task, too many keep noise. The best views share exactly the task bits (the InfoMin principle). The evidence is a family of knobs (patch offset, colour channels, blur, crop, jitter) that trace the U, a recipe (InfoMin Aug) that reaches 73.0% on ImageNet, a toy world (Colorful-Moving-MNIST) that shows the best views depend on the task, and a semi-supervised, invertible view generator that finds the sweet spot for colour channels.

The numbers

QuantityValueWhere
Patch-offset sweet spotd = 128 (patches 64 by 64)Fig. 3, STL-10 and CIFAR-10
Blur: max shared bits, sweet spotσ = 0.7, then 1.0Fig. 9, appendix
Crop low-area bound, sweet spotc = 0.2 (not 0.08)Fig. 5, Fig. 13
Color jitter strength, sweet spotx = 1.0Fig. 5, Fig. 14
ImageNet top-1, InfoMin, 200 and 800 epochs70.1% and 73.0%Table 1
SimCLR top-1, 1000 epochs69.3%Table 1
Ladder: RA-CJ-Blur, + JigSaw, + MLP head63.6, 65.9, 67.3 (100 epochs)Appendix
COCO box AP, InfoMin against supervised40.6 against 39.7 (1x)Table 2
VOC AP75, InfoMin against supervised64.6 against 58.8Table 3
Digit error: digit shared alone, against digit, position and background shared16.8% against 88.8%Table 4
STL-10: semi-supervised views, RGB and YDbDr86.0% and 87.0%Table 5
View-generator and critic learning rates2e-4 and 6e-4 (Adam)Appendix, unsupervised game
MoCo-style runs for single-augmentation plots65,536 negatives, 100 epochs, batch 256Appendix

The terms

TermPlain meaning
ViewOne version of a data item: a crop, a channel, a patch, a sound clip.
I(v1; v2)The bits the two views share.
INCEA lower bound on that, from the contrastive loss: log K minus the loss.
Sufficient, minimalAn encoder that loses nothing the game needs, and keeps nothing else.
Optimal representationThe minimal sufficient statistic of the label y.
InfoMax, InfoMinCapture all information, or share only the task’s bits.
Linear readoutFreeze the encoder, train one linear layer on a task.
ShortcutA factor with many bits that lets the model match pairs without learning the task.

A checklist for choosing views

  1. Write the downstream task down. Name its factors and the nuisance factors.
  2. Make sure each view keeps every task factor. If a view cannot predict the label, stop.
  3. List what the views still share beyond the task factors. Each item is a possible shortcut.
  4. Remove the shared nuisance: jitter colours, crop harder, blur, shuffle, drop channels.
  5. Sweep one knob at a time. Record the downstream score and an INCE estimate with a fixed K and architecture.
  6. Stop at the peak. If the score never peaks, the knob is too weak; if it only falls, the views lose task bits.
  7. Check INCE against log K. If it sits at the ceiling, add negatives before you compare.
  8. With a few labels, consider learning the views with the label term, and check the spread over runs.

The core as code

Two pieces matter: the InfoNCE loss with its bound, and one step of the semi-supervised view-learning game. The coupling block is a sketch of an invertible pixel-wise map, our own simple choice; the paper uses flow blocks (VP and NVP) whose exact form is in its Fig. 11.

python
import math, torch, torch.nn as nn, torch.nn.functional as F

def info_nce(z1, z2, tau=0.15):
    # z1, z2: (B, D) codes of the two views. Row i of z2 is the partner of row i of z1.
    z1, z2 = F.normalize(z1, dim=1), F.normalize(z2, dim=1)
    logits = z1 @ z2.t() / tau                 # (B, B): h(v1_i, v2_j) = cos / tau
    target = torch.arange(len(z1))             # the diagonal is the positive pair
    loss = F.cross_entropy(logits, target)     # L_NCE, in nats; B candidates, so K = B
    i_nce = math.log(len(z1)) - loss           # I(v1; v2) >= log K - L_NCE
    return loss, i_nce

class Coupling(nn.Module):               # an invertible, pixel-wise block (1x1 convs)
    def __init__(self, c1, c2):
        super().__init__()
        self.f = nn.Sequential(nn.Conv2d(c2, 16, 1), nn.ReLU(), nn.Conv2d(16, c1, 1))
    def forward(self, x1, x2):               # inverse: x1 = y1 - f(x2)
        return x1 + self.f(x2), x2

def view_step(g, f1, f2, c1, c2, x_unl, x_lab, y, opt_g, opt_f):
    # x_unl: (B, 3, H, W) unlabeled batch; x_lab: (b, 3, H, W) labeled batch; y: (b,)
    # 1) encoders ascend I_NCE on views from a frozen g (minimise the loss)
    with torch.no_grad(): xh = g(x_unl)
    loss_f, _ = info_nce(f1(xh[:, :1]), f2(xh[:, 1:]))    # views: (B,1,H,W) and (B,2,H,W)
    opt_f.zero_grad(); loss_f.backward(); opt_f.step()
    # 2) generator descends I_NCE + label terms (Eq. 3)
    xh = g(x_unl); _, i_nce = info_nce(f1(xh[:, :1]), f2(xh[:, 1:]))
    xl = g(x_lab)
    ce = F.cross_entropy(c1(xl[:, :1]), y) + F.cross_entropy(c2(xl[:, 1:]), y)
    opt_g.zero_grad(); (i_nce + ce).backward(); opt_g.step()    # lower shared bits, keep label bits

Where it sits in the field

This paper sits between two ideas from its own related work. One is mutual-information maximisation (Deep InfoMax, CPC), the InfoMax principle. The other is the wave of InstDis, MoCo and SimCLR, which found good augmentations mostly by trial. The paper adds a rule for choosing views and a way to test it. The same question returns in multimodal learning, where a photo and its caption are two views, and what they share is what the model can align.

Keep going:

The takeaway. What two views share is what a contrastive model learns. Choose views that share the task’s bits and nothing else, check the sweet spot with a sweep, and treat every shared nuisance as a possible shortcut. The views are a design decision, as important as the data and the labels.

Credit. The paper is “What Makes for Good Views for Contrastive Learning?” by Yonglong Tian (MIT CSAIL), Chen Sun (Google, Brown University), Ben Poole (Google Research), Dilip Krishnan (Google Research), Cordelia Schmid (Google Research) and Phillip Isola (MIT CSAIL), NeurIPS 2020. Read it on arXiv 2005.10243; the authors’ project page is hobbitlong.github.io/InfoMin. This page re-teaches the paper in our own words and devices, and toy numbers are labelled as toys wherever they appear.

You build views for a new sensor pair and the downstream score keeps falling as you strengthen the augmentation. What does the paper suggest checking first?

Now press Teach and explain from memory: why the views are the teacher, what InfoNCE counts, the three regimes, how one factor swallows the rest, and how the view generator finds the sweet spot. Then go back to the patches at the top, slide the offset, and say which regime you are in at each setting.

What Makes for Good Views for Contrastive Learning?
Read the paper · Back to MMAI Lecture 7: Alignment · Back to microLearning