Hide three quarters of a photo, show the encoder only the rest, and let a small decoder repaint the gaps. That one trick trains a vision model about 3 times faster.
Learn how a vision model teaches itself to see by filling in missing pieces of pictures, with no labels at all.
Choose who reads the picture, then drag the masking ratio. Watch how many patches the encoder must read. Then we build, chapter by chapter, everything underneath: the patches, the shuffle, the lopsided encoder and decoder, the pixel loss, and every test the authors ran.
You need a rough idea of what a neural network is (adjustable arithmetic that learns from examples) and what an image is (a grid of numbers). We build the rest from zero.
The assigned reading for Lecture 8 of MIT MMAI (Multimodal AI, Spring 2026, Paul Liang). Paper: Masked Autoencoders Are Scalable Vision Learners, by Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár and Ross Girshick (Facebook AI Research). Every idea is re-taught here in our own words and devices; the numbers are theirs.
The picture is a made-up 14 × 14 patch scene (196 patches, the paper’s count for a 224 × 224 image cut into 16 × 16 patches). The counts are exact: the encoder keeps floor(196 × (1 − ratio)) patches, and attention compares every kept patch with every other. The paper’s own measurement at 75% is 3.3 times fewer training FLOPs (Table 1c), because the decoder still reads all 196.
Chapter 0
See why a method that worked on words did not carry over to images, and what three small changes fixed that
Imagine a friend covers a few words of a sentence with sticky notes and asks you to read it anyway. “The cat sat on the ___.” You say “mat” at once. To fill one gap well, you had to understand the sentence.
That game is the engine behind modern language models. Nobody writes the answer on a label. The text itself holds the answer: hide a piece, ask the model to guess it, and check against what was hidden. This is self-supervised learning: the data supplies its own exam, so no person has to label anything.
Researchers wanted the same free lesson for pictures. Cover part of a photo, ask a model to repaint it, and the model should learn what photos contain. The idea was old, and the language version, called BERT, had been a huge success. Yet vision methods lagged behind. This paper, from Facebook AI Research, asks why, and answers with a method called the masked autoencoder, or MAE.
An autoencoder is a model with two halves. The encoder squeezes an input into a short summary, a list of numbers called the latent representation. The decoder takes that summary and tries to rebuild the input. If the rebuild is good, the summary must hold what matters. Think of describing a painting over the phone so that a friend can redraw it.
A masked autoencoder adds a twist. Part of the input is hidden before the encoder sees it, and the decoder must still rebuild the whole thing. A denoising autoencoder is the general family: damage the input, then repair it. Masking is one kind of damage.
The authors name three differences between language and vision. Each one shapes a design choice you will meet in this lesson.
One: the architecture. BERT is built on the transformer, a model that reads a list of pieces and lets every piece look at every other. Images used to belong to convolutional networks, which slide small filters over a regular grid. A grid has no easy place for a “this piece is missing” marker. The Vision Transformer (ViT) removed this obstacle by turning an image into a list of pieces. Chapter 1 shows how.
Two: information density. A sentence is packed. Each word carries meaning, so hiding even one forces real understanding. A photo is the opposite. Neighbouring pixels look alike, so a hidden patch can be guessed from the patches around it, with no grasp of objects at all. The paper calls this spatial redundancy. If you hide only 15% of a photo, as BERT hides 15% of words, the task is too easy. The model learns to smear colours, not to see.
Three: what the decoder predicts. In BERT the decoder predicts a word, which is already meaning. In vision the MAE decoder predicts pixels, which are low-level. So the decoder does more of the “low-level work”, and its design decides how abstract the encoder’s summary becomes. Chapter 4 returns to this.
Try the second difference yourself. The device below hides patches of a picture, then fills each hole by copying the average colour of the nearest visible patches. That is a model with no understanding at all.
Left: the picture with patches hidden. Right: each hole filled with the average colour of the nearest visible patches. Drag the ratio and watch the copy go from nearly perfect to useless.
Illustrative: a made-up 56 × 56 pixel scene, not an ImageNet photo, and a toy filler, not the paper’s model. The error is the average gap between the true and filled pixel values (0 to 255 scale) over hidden patches only, for one fixed random mask per ratio.
At 15% hidden, the copy is close. A hole is surrounded by visible patches, so averaging them works. At 75% hidden, most holes sit deep inside a blank region and the filler has nothing to copy. A good filler must now know that the orange shape is round and the lower band is grass. That is the point. A hard task forces the model to learn something worth keeping.
The authors state the claim as a trade. Hide a very large share of the patches, say 75%. Let the big encoder read only the visible 25%. Let a small decoder do the repainting. The task is harder, so the encoder learns more. The encoder sees fewer patches, so training costs far less. The paper reports a speedup of 3 times or more, and a vanilla ViT-Huge model that reaches 87.8% accuracy on ImageNet-1K using only that dataset.
Take the claims one at a time. ImageNet-1K is a set of about 1.3 million labelled photos in 1,000 classes, the standard test for vision models. Accuracy here means the share of test photos whose class the model names correctly. The pre-training step uses the photos without their labels. Labels appear only when the authors test the result.
Pre-training has no exam of its own. A model that repaints well might still have learned nothing useful. So the authors test the encoder on a real job, in two ways that you will see in every table.
Linear probing freezes the encoder and trains one thin layer on top, a straight-line classifier, to read its summaries. If the summaries already separate cats from cars, a straight line is enough. It tests how good the summary is as it stands. Fine-tuning lets every weight of the encoder keep learning on labelled photos. It tests how good a starting point the encoder is. The two scores often disagree, and the disagreement teaches something. Chapter 7 returns to it.
Keep one more number in mind. A plain ViT-Large trained from scratch with labels reaches 82.5% on ImageNet-1K in the authors’ careful recipe. Any pre-training must beat that to earn its cost.
This lesson follows the paper’s own order, one mechanism per chapter. It is the assigned reading for Lecture 8, Large Multimodal Models, where MAE appears as a way to learn from images without labels.
Chapter 1
Turn an image into a short list of vectors that a transformer can read, and learn the shapes you will need all lesson
Think of a jigsaw puzzle. You do not reason about every grain of the cardboard. You reason about pieces, each with a picture on it. A transformer needs the same thing from a photo: a list of pieces. This chapter builds that list, and fixes the shapes that the rest of the lesson reuses.
A colour photo is a grid of pixels. Each pixel holds three numbers: how much red, green and blue. The paper works with images of 224 × 224 pixels. So one image is 224 × 224 × 3 numbers, which is 150,528 of them. A tensor is just a name for such a table. We write its shape as a list of sizes, for example [3, 224, 224] for three colour channels of 224 by 224.
A transformer cannot take 150,528 separate inputs. Attention compares every input with every other, and the work grows with the square of the count. That is about 22 billion pairs. We need far fewer, bigger pieces.
The Vision Transformer (ViT) cuts the image into a regular grid of square, non-overlapping patches. With patches 16 pixels wide, a 224-pixel side holds 14 patches (224 ÷ 16 = 14). The grid is 14 × 14, so there are 196 patches. Each patch holds 16 × 16 × 3 = 768 numbers. Stretch those 768 numbers out in a line and you have one flattened patch.
Worked example with the paper’s numbers. H = W = 224, P = 16, C = 3. Then N = 14 × 14 = 196, and dpatch = 16 × 16 × 3 = 768. The image becomes a table of shape [196, 768]. The device below lets you change P and see the trade.
Choose a patch size. Then tap any patch in the picture. The strip underneath is that patch, flattened into one line of numbers (brightness stands for value).
The 224 × 224 picture is the made-up scene from the hero, enlarged four times so every patch size divides it. Counts are exact arithmetic; the strip bins the flattened values into at most 96 cells. The paper uses 16 × 16 for ViT-B and ViT-L and 14 × 14 for ViT-H.
Small patches give many pieces, each with few numbers. At P = 8 there are 784 patches, and attention compares 614,656 pairs. Big patches give few pieces with many numbers each. At P = 32 there are 49 patches but 3,072 numbers per patch. P = 16 sits in the middle. This is the “image is worth 16×16 words” idea of the ViT paper.
A flattened patch of 768 numbers is raw pixels. The transformer wants vectors in its own working size. For ViT-Large that size is 1,024. A linear projection converts one into the other. It is a single matrix, learned during training, of shape [768, 1024]. Multiply each flattened patch by it. The result is a token: one vector standing for one patch. Now the image is a table of shape [196, 1024].
One problem remains. A transformer treats its input as a bag. If you shuffle the patches, it cannot tell. But position matters: sky sits above grass. So we add a positional embedding, a vector that says where the patch came from. Think of a numbered badge pinned to each piece of the jigsaw. The paper uses the fixed sine-cosine version: each position gets a vector of sines and cosines at many speeds, like a bank of clocks with hands at different rates, which gives every grid cell a unique pattern. Nothing about it is learned.
The tokens now pass through a stack of transformer blocks. Each block has two parts. The first is self-attention: every token looks at every other token and mixes in the ones that seem relevant, the way you glance at the whole table before you decide which puzzle piece to pick up. The second is an MLP (a small two-layer network) that processes each token on its own. Both parts are wrapped in LayerNorm, a step that rescales each vector so training stays stable.
ViT-Large has 24 such blocks at width 1,024. Its cost has one important property: attention compares all tokens with all tokens, so fewer tokens means much less work. Chapter 3 uses this. The output of the last block is still shape [196, 1024]: one improved vector per patch. The ViT also adds a class token, a spare learnable vector that collects a summary of the whole image for classification. The paper appends a dummy token for this and notes it works about as well with average pooling instead.
Chapter 2
Shuffle the patches, keep the first few, and remember how to put everything back
You hold a deck of 196 cards, one for each patch. Your job is to pull out 49 of them, chosen with no pattern at all, and then be able to put the deck back exactly as it was. The simplest way is to shuffle the deck, deal off the top 49, and write down the shuffle. That is the whole masking step.
The paper names its rule random sampling. Take the patches of an image and pick a subset to keep, without replacement (a patch cannot be picked twice) and with a uniform distribution (every patch is equally likely). The rest are masked, which here means removed. The share removed is the masking ratio. The default is 75%.
Check it against the paper. At ρ = 0.75, K = floor(196 × 0.25) = floor(49) = 49. At ρ = 0.80, K = floor(39.2) = 39. That is exactly the “39 out of 196 patches” in the paper’s Figure 2. For comparison, BERT hides about 15% of words, and earlier vision work hid 20% to 50% of patches. MAE goes much higher, and the paper’s own experiments say the sweet spot is 75%.
How do you pick K tokens at random in code, without special sparse operations? The paper’s answer is plain. First make a token for every patch, with its position already added. Then shuffle the whole list and cut off the end. The front K tokens are your sample, and every possible choice of K tokens is equally likely.
In practice the shuffle uses sorting. Give every token a random number between 0 and 1 (its noise). Sort the tokens by their noise. The sorted order of indices is the shuffle; call it ids_shuffle. Keep the first K entries. To undo the shuffle later, you need the inverse: for each original index, where did it land? Sorting the shuffle order again gives that. Call it ids_restore. Step through it below on twelve tokens.
Press Next step to move through the five steps. Choose a masking ratio first or part-way through; the kept set changes with it.
Illustrative: twelve tokens instead of 196, with fixed made-up noise values (shown as whole numbers, noise × 100). The method is the paper’s: sort by random noise, keep the first K, and sort the order again to get the restore list.
Look at the last step. The restore list is small and cheap to store. It lets the decoder put every vector back in its true place, which Chapter 4 does. Nothing in this procedure needs a special kind of sparse layer. It is a sort, a slice and a gather. The paper notes the shuffling cost is negligible.
Follow the real sizes for ViT-Large at 75%. Patch tokens with positions: [196, 1024]. Noise: [196]. Kept ids: the first 49 entries of ids_shuffle. Kept tokens, after a gather: [49, 1024]. In a batch, add a leading batch size B to each shape, and each image gets its own noise. The mask is therefore different in every image and every training step.
That last fact matters. The paper points out that fresh masks generate new training samples even without any other data augmentation. The same photo is a new puzzle each time. Chapter 6 shows that MAE works with almost no augmentation, which contrasts with contrastive methods that rely heavily on it.
Random sampling does two jobs. First, with a high ratio it removes most of the redundancy from Chapter 0, so a hole cannot be filled by stretching a line or a texture from its neighbours. Second, the uniform draw prevents a centre bias, where masked patches would pile up near the middle of the image. Both are about making the task honest.
Compare other patterns in your head. If the mask removed one large square block, the model would face a hole with no visible patch inside it, a much harder job. If the mask kept one patch in every four in a regular grid, the model could copy nearby patches, a much easier job. The paper tests both and finds random works best. The numbers are in Chapter 6.
Chapter 3
Run the big model on the visible patches only, and count what that saves
Picture a reader asked to summarise a book from a quarter of its pages. A careless setup hands the reader the whole book, with the missing pages replaced by blank sheets, and the reader must still turn every sheet. A better setup binds only the real pages, so the reader turns fewer sheets. MAE does the second thing. This chapter shows why that choice is cheaper, and also more accurate.
The MAE encoder is an ordinary ViT. The paper’s ablation model is ViT-Large: 24 transformer blocks, width 1,024. It uses the patch projection and the positional embeddings from Chapter 1. The one change is the input. It gets only the visible patches. At a 75% ratio that is 49 tokens, shape [49, 1024], plus one dummy class token. Masked patches are not in the list. No placeholder stands in for them.
Because the positional embeddings were added before the shuffle, each visible token still knows where it came from. The encoder never sees the shuffle at all. After the last block, a final LayerNorm runs, and a linear layer maps the width from 1,024 down to the decoder’s 512. The output is [49, 512] (plus the class slot), ready for Chapter 4.
A transformer layer spends effort in two places. The linear layers (the attention projections and the MLP) do work proportional to the number of tokens n. The attention comparisons do work proportional to n squared. In rough terms, for one layer with width d, counting each multiply and add as two operations:
Work it for ViT-Large (d = 1,024) with all 196 tokens. The linear part is 24 × 196 × 1,0242 = 4.93 billion operations. The attention part is 4 × 1962 × 1,024 = 0.16 billion. One layer costs 5.09 billion, and 24 layers cost about 122 billion. Now use only 49 tokens. The linear part is 1.23 billion, the attention part 0.01 billion. One layer costs 1.24 billion, and 24 layers cost 29.8 billion. The encoder alone is 4.1 times cheaper.
The decoder adds back some cost. It has 8 blocks at width 512 and reads all 196 tokens: about 10.5 billion operations. So MAE totals roughly 40 billion operations, against 133 billion for an encoder that carries the mask tokens. That is 3.3 times fewer. The paper reports exactly 3.3 times fewer training FLOPs (Table 1c). Try your own settings.
Two ways to build the same ViT-Large model. Top bar: mask tokens go through the encoder. Bottom bar: MAE, visible tokens only. Drag the ratio and the decoder depth.
An estimate for one forward pass of one image, from the formula above. Encoder: ViT-L, 24 blocks, width 1,024. Decoder: width 512, all 196 tokens. It ignores LayerNorm, the patch projection and the loss. The paper’s measured figure is 3.3 times fewer FLOPs at 75% with an 8-block decoder (Table 1c).
Slide the ratio to 90% and the gap widens to about 6 times, because attention shrinks with the square. Push the decoder to 12 blocks and the decoder eats a larger share. This is why the paper keeps the decoder narrow: it processes every token, so every block counts.
Compute is only half the story. Suppose the encoder did carry mask tokens. During training, three quarters of its input would be a learned “nothing here” vector. At test time, you give it a real, uncorrupted image with no mask tokens at all. The encoder sees a kind of input it was never trained on. This mismatch is a train and deploy gap.
The failure shows up in the numbers. Symptom: the summaries are worse. Measurement (Table 1c, ViT-L, 75% ratio): with mask tokens in the encoder, linear probing reaches 59.6% and fine-tuning 84.2%. Without them, 73.5% and 84.9%. That is a 14-point gap for the straight-line classifier. Fix: keep mask tokens out of the encoder, so it always sees real patches.
FLOPs are a count; time is what you pay for. Table 2 reports 800-epoch training on 128 TPU-v3 cores. The ViT-L with mask tokens takes 42.4 hours. MAE takes 15.4 hours, a 2.8 times speedup. With a one-block decoder it takes 11.6 hours (3.7 times). For ViT-Huge, the mask-token version is estimated at 119.6 hours (the paper trained ten epochs and extrapolated), against 34.5 hours for MAE, a 3.5 times speedup. With a one-block decoder, 29.3 hours, or 4.1 times. Memory shrinks too, which allows larger batches or larger models.
Chapter 4
Bring back the blanks as mask tokens, put every vector in its place, and predict pixels
The encoder has studied the visible patches and written its notes: 49 vectors. Now a painter must use those notes to fill a canvas of 196 patches. Where a patch was visible, the notes describe it. Where a patch was hidden, the painter has only a blank slot and the knowledge of where the slot sits. That painter is the decoder.
The decoder’s input is the full set of 196 tokens. It has two kinds. The first kind is the 49 encoded visible tokens, now of width 512. The second kind is the mask token: one shared, learned vector of width 512, written [M]. It says “a patch is missing here; please predict it.” The same vector is copied 147 times, once for each hidden patch.
The tokens are first in the wrong order: 49 visible ones, then 147 blanks. We must put each one back where it belongs. That is the unshuffle, and it uses the restore list from Chapter 2. A small example with six tokens and a 50% ratio shows how.
Say the noise values are 0.8, 0.1, 0.5, 0.9, 0.3, 0.6 for tokens 0 to 5. Sorting from low to high gives ids_shuffle = [1, 4, 2, 5, 0, 3]. The encoder keeps the first three: tokens 1, 4 and 2. Sorting that list again gives ids_restore = [4, 0, 2, 5, 1, 3]: token 0 sat at position 4, token 1 at position 0, and so on. The decoder’s list is [e1, e4, e2, M, M, M]. For each slot i, take the entry at position ids_restore[i]. Slot 0 takes position 4, which is M. Slot 1 takes position 0, which is e1. The result is [M, e1, e2, M, e4, M]. The encoded tokens are home, and the blanks sit where the hidden patches were.
Next, the decoder adds positional embeddings to all tokens, again the sine-cosine kind. The paper explains why: without them, mask tokens would have no information about their location in the image. Every [M] is the same vector. Only the position tells the decoder which patch to paint.
The tokens pass through another stack of transformer blocks. The default has 8 blocks of width 512. Through attention, each mask token looks at the visible tokens and at the other mask tokens, and gathers evidence about what belongs in its slot. A single block is the minimum that lets information flow from visible tokens to mask tokens, and the paper shows even that can work well for fine-tuning.
The last layer is a linear projection whose output size equals the number of pixel values in a patch: 16 × 16 × 3 = 768. So each of the 196 tokens becomes a vector of 768 numbers, a predicted patch. The output table has shape [196, 768]. Reshape each row back into a 16 × 16 × 3 square and tile the squares in grid order. You have a [3, 224, 224] picture again. Chapter 5 says how it is scored.
# one MAE training forward pass, ViT-L, 75% masking, batch size B x = patchify(img) # [B, 196, 768] pixels, one row per patch tok = embed(x) + pos # [B, 196, 1024] linear projection + sine-cosine noise = rand(B, 196) ids_shuffle = argsort(noise) # [B, 196] ids_restore = argsort(ids_shuffle) # [B, 196] keep = ids_shuffle[:, :49] vis = gather(tok, keep) # [B, 49, 1024] z = encoder(vis) # [B, 49, 1024] 24 blocks, no mask tokens z = dec_embed(z) # [B, 49, 512] full = concat(z, M.repeat(147)) # [B, 196, 512] M is one learned vector full = gather(full, ids_restore) + dec_pos # unshuffle, add positions pred = dec_head(decoder(full)) # [B, 196, 768] 8 blocks, then linear
Recall Chapter 0, difference three. The decoder predicts pixels, which sit at a lower level of meaning than a class name. The last layers of any autoencoder specialise in the exact job of reconstruction. If those layers were part of the encoder, its final summary would be tied to pixel detail. A reasonably deep decoder takes on that specialisation and leaves the encoder’s summary more abstract. The ablation below measures this.
The decoder is used only in pre-training. It is independent of the encoder’s design, so the authors make it small. The default has under 10% of the encoder’s compute per token: 9% of the FLOPs per token of ViT-L. After pre-training the decoder is discarded. Recognition tasks use the encoder alone, fed whole, uncorrupted images with all 196 patches.
Choose to vary the decoder’s depth or its width. Each pair of bars is one trained model: fine-tuning accuracy (warm) and linear probing accuracy (teal). Tap a pair’s setting with the slider.
Numbers are the paper’s Table 1a and 1b: ViT-L/16, ImageNet-1K, 800 epochs, 75% masking. The vertical axis starts at 60% so that small gaps are visible. The default (8 blocks, 512 channels) is marked with a dot.
Reading the chart: fine-tuning barely moves. It sits near 84.9% for most settings, so the decoder’s size hardly matters there. The encoder’s last layers can adjust during fine-tuning. Linear probing is different. It climbs from 65.5% with one block to 73.5% with eight, an 8-point gain. The straight-line classifier needs the encoder’s summary to be ready-made, and a deeper decoder helps keep it clean. Width shows a mild version: 512 is as good as any larger size.
Chapter 5
Define the pixel target, normalise it patch by patch, and compute the loss on the masked patches alone
A student redraws a landscape from memory and the teacher grades it. What does the teacher compare, and which parts? Compare every pixel of the redraw with the original? Or only the parts the student could not see? The answer shapes what the model learns, and the paper makes two choices here that look small and matter.
The target is simple: the original pixel values. The decoder outputs a predicted patch of 768 numbers for each token. The loss is the mean squared error (MSE) between prediction and original. For each pixel, subtract, square (so that both too-high and too-low count, and big misses count extra), then average.
The loss is computed on masked patches only, the same choice BERT makes for its hidden words. Visible patches do appear in the decoder’s output, but nobody scores them. The paper’s own figures say so: in Figure 2 the output on visible patches looks worse, because it was never graded, and the authors choose not to paste the real patches back so that the method’s behaviour stays honest.
The paper gives its reason in a footnote, and it is practical: grading all pixels lowers accuracy slightly, by about 0.5%. A visible patch is already given to the model, so a score on it adds easy, nearly-solved terms to the average and dilutes the hard ones. Classical denoising autoencoders grade everything; MAE does not.
The second choice is a variant of the target. Instead of raw pixel values, the model predicts normalised pixels. For each patch, compute the mean μ and the standard deviation σ of its pixel values (the standard deviation measures how spread out the values are). Then replace every pixel x by (x − μ) / σ. A normalised patch always has mean 0 and spread 1, whatever the lighting.
A worked example with four pixel values in one patch: 0.2, 0.4, 0.6, 0.8. The mean is 0.5. The squared gaps are 0.09, 0.01, 0.01, 0.09; their average is 0.05, so σ = 0.224. Normalised values: −1.342, −0.447, 0.447, 1.342. Suppose the decoder predicts −1.0, −0.5, 0.5, 1.2. The gaps are 0.342, −0.053, 0.053 and −0.142. Squared: 0.117, 0.003, 0.003, 0.020. The mean is 0.0357. That is the loss for this patch.
Why does this help? A bright flat patch of sky has tiny variation, and a patch with sharp edges has large variation. With raw pixels, the squared error is dominated by high-contrast patches, and flat patches barely count. Normalising makes every patch matter equally and sharpens local contrast. The device below shows it. Try the two targets on the same predictions.
Eight patches of four pixels. Blue bars: the true values. Pink dots: the decoder’s guess. Purple patches are hidden. The bars underneath are each patch’s loss. Switch the target and what is scored.
Illustrative toy numbers, not from the paper. Visible patches are guessed almost right (95% of their contrast); hidden ones are guessed blurry (20% of their contrast). Eight patches at 75% hidden means six are hidden. The paper’s own result: normalised pixels beat raw pixels, 85.4% against 84.9% fine-tuning (Table 1d).
With raw pixels, the loss comes almost entirely from the high-contrast patches. The flat sky patches add almost nothing. With normalised pixels, a blurry guess costs about the same on every patch, so flat regions get real training pressure too. Also try the second control: scoring the visible patches adds near-zero terms, and the average falls, with the same prediction quality on the hidden patches.
The paper tests two other targets. In the first, it predicts the 96 largest PCA coefficients of each patch (PCA finds the main directions in which patches vary, keeping smooth shapes and dropping fine detail). Result: worse (84.6% fine-tuning, 72.3% linear probing), which suggests the fine, high-frequency detail is useful. In the second, it predicts discrete tokens from a pre-trained image tokenizer, a dVAE, as BEiT does. Fine-tuning reaches 85.3%, about the same as normalised pixels (85.4%), but linear probing falls to 71.6%, and the tokenizer needs a whole extra training stage on 250 million images. Its encoder costs 40% of ViT-L’s FLOPs. Table 7 shows that across ImageNet, COCO and ADE20K the gap between tokens and normalised pixels is statistically insignificant. Pixels win by being simple.
Chapter 6
Read the ablation tables one choice at a time: how much to mask, how to pick, how long to train
A cook tests a recipe by changing one thing at a time: less salt, more heat, a longer bake. If the dish gets worse, that ingredient mattered. The paper does the same. It trains a ViT-Large with a default setting, changes one choice, and measures again. This is an ablation. The default is: decoder 8 blocks and 512 wide, unnormalised pixel target, random-resized-crop augmentation, 75% masking, 800 epochs. An epoch is one pass over the whole training set. Every score below is ImageNet-1K validation accuracy.
Figure 5 varies the masking ratio. The best value is 75%, for both ways of grading. This is far above BERT’s 15% and above the 20% to 50% of earlier vision work. The two grades behave differently, though. Linear probing climbs steadily as the ratio rises to its sweet spot: at a 10% ratio it scores 54.6%, at 75% it scores 73.5%, a gap of nearly 20 points. Fine-tuning is much less sensitive. Any ratio from 40% to 80% works well, and every fine-tuning result in the figure beats the 82.5% of training from scratch.
The reason, in the authors’ words, is that the model must infer missing patches by understanding the whole scene (Figure 4 shows reconstructions that differ from the original but look plausible), which they link to useful features. Chapter 0’s device showed the mechanism: more hiding, less copying.
Table 1f compares three ways. Random picks patches one by one at uniform. Block-wise (from BEiT) removes large connected blocks. Grid-wise keeps one patch in every group of four, in a regular pattern. Pick a strategy below and see the mask, the paper’s scores, and a toy measure of how hard it is to fill.
Choose a pattern. Purple squares are hidden, blue squares are visible. The bars give the paper’s accuracy for that pattern.
Accuracy numbers are the paper’s Table 1f (ViT-L/16). The mask shapes are redrawn here by simple rules (seeded random blocks of 3 to 6 patches; every second row and column for the grid), not copied from Figure 6. The “toy copy error” reuses Chapter 0’s neighbour-copy filler on the made-up scene, so it only suggests difficulty.
Random wins: 84.9% fine-tuning and 73.5% linear probing at 75%. Block-wise at 50% is a little worse (83.9, 72.3). At 75% it drops to 82.8 and 63.9: a large hole contains no visible patch, the task becomes too hard, the training loss is higher and reconstructions turn blurry. Grid-wise gives 84.0 and 66.0. It is too easy, with a lower training loss and sharper pictures, but the representation is worse. Easy tasks teach little. Random also allows the highest ratio, which gives the biggest speedup.
Data augmentation means altering each training image in a random way (cropping, flipping, shifting colours) so the model cannot memorise it. Contrastive methods rely on it heavily; one report cited in the paper finds that cropping-only reduces BYOL by 13% and SimCLR by 28%. MAE needs far less, because random masking is already a new puzzle each time.
| Augmentation (Table 1e) | Fine-tuning | Linear probing |
|---|---|---|
| None (centre crop only) | 84.0 | 65.7 |
| Crop, fixed size | 84.7 | 73.1 |
| Crop, random size (default) | 84.9 | 73.5 |
| Crop plus colour jitter | 84.3 | 71.9 |
Even with no augmentation, MAE reaches 84.0% when fine-tuned. Adding colour jitter makes things worse. A contrastive method would collapse without augmentation, because its two views of one image would be identical and a trivial solution would satisfy it.
Figure 7 trains for 100, 200, 400, 800 and 1,600 epochs. Fine-tuning accuracy goes 82.3, 83.3, 84.3, 84.9, 85.1. Linear probing goes 57.3, 64.4, 69.7, 73.5, 75.1. There is no sign of saturation even at 1,600 epochs for linear probing. A contrastive method like MoCo v3 saturates at 300 epochs for ViT-L. One reason may be that the MAE encoder sees only 25% of the patches per epoch, while a two-crop contrastive method sees 200%. Even 1,600 epochs of MAE is cheaper per epoch: the paper reports 31 hours for ViT-L on 128 TPU-v3 cores against 36 hours for MoCo v3 at 300 epochs.
Put the five results together. A reader who wants to replicate the method should keep: random masking at 75%, a shallow decoder with 8 blocks, pixel targets with per-patch normalisation, cropping only, and as many epochs as the budget allows.
Chapter 7
Compare the encoder with other methods, with supervised training and on new tasks, then name what the paper does not show
You trained a model on pictures with no labels. Does it pay off? A fair test has four parts: compare with other ways of pre-training, compare with plain supervised training, try the encoder on jobs it never saw, and look for where it breaks. This chapter reads the paper’s tables in that order. The numbers are the authors’.
Table 3 fine-tunes each encoder end to end on ImageNet-1K and reports top-1 accuracy: the share of validation photos for which the model’s single best guess is right. For ViT-Base all methods land close together: scratch 82.3, DINO 82.8, MoCo v3 83.2, BEiT 83.2, MAE 83.6. The gaps grow with size. ViT-Large: scratch 82.6, MoCo v3 84.1, BEiT 85.2, MAE 85.9. The paper reads this as a sign that the main challenge for big models is overfitting, and that pre-training reduces it.
MAE then scales. ViT-Huge reaches 86.9% at 224 pixels. Fine-tuned at 448 pixels it reaches 87.8%, the best published result using only ImageNet-1K data. The previous best was 87.1%, at 512 pixels, with an advanced network. Note that MAE uses a vanilla ViT. Against BEiT, MAE is more accurate while being simpler (no dVAE stage) and 3.5 times faster per epoch. Against MoCo v3, MAE trains for 1,600 epochs in 31 hours on 128 TPU-v3 cores, while MoCo v3 needs 36 hours for 300 epochs.
Figure 8 plots accuracy against model size. Supervised training on ImageNet-1K alone saturates as models grow: the original ViT paper saw ViT-L get worse, and even the authors’ improved recipe flattens. MAE pre-training, on the same images and with no labels, keeps rising. The gain over training from scratch is bigger for bigger models: +1.3 points at ViT-B (83.6 against 82.3), +3.3 at ViT-L (85.9 against 82.6), +3.8 at ViT-H (86.9 against 83.1). That follows the pattern the original ViT authors found with supervised pre-training on the 300-million-image JFT dataset. It supports the paper’s title: the method scales.
A detail worth knowing. Training ViT-L and ViT-H from scratch with labels was unstable: the loss often blew up into an invalid number that ruined every step after it. The authors built a careful recipe (weight decay 0.3, batch size 4,096, long warmup) before the baseline would finish. So the 82.6 and 83.1 baselines are strong, not strawmen.
MAE is not the best in straight-line probing. The paper says that MoCo v3 has higher linear probing accuracy. But linear probing and fine-tuning are largely uncorrelated (Table 1), and probing forces the model to be good at a very narrow job: separating classes with a straight line. Deep learning’s strength is non-linear features. So the authors test the middle: partial fine-tuning, which tunes the last few blocks and freezes the rest. Tuning only one block lifts MAE from 73.5% to 81.0%. Tuning just the MLP half of the last block gives 79.1%. Tuning four to six blocks nears full fine-tuning (84.9%). For every number of blocks tuned, from one up, MAE beats MoCo v3, by 2.6 points at four blocks.
Pick a benchmark. Each group is a pre-training method; the dark bar is ViT-Large, the pale bar ViT-Base. Find where MAE (warm) sits.
All numbers are from the paper: ImageNet top-1 (%), COCO box AP, ADE20K mIoU, and ViT-L accuracy on ImageNet when only the stated part of the encoder is tuned. A missing bar means the paper does not report that size for that method. Bars do not start at zero; the axis range is printed on each chart.
The encoder is fine-tuned inside a standard detector on COCO (finding and outlining objects; box AP scores the boxes, mask AP the outlines) and a standard segmenter on ADE20K (labelling every pixel as one of many scene classes; mIoU scores the overlap). In Table 4, ViT-L box AP is 53.3 for MAE against 49.3 for supervised pre-training, 4.0 points better. In Table 5, mIoU is 53.6 for MAE against 49.9 supervised, 3.7 points. Pixel-based MAE equals or beats token-based BEiT (53.3 box AP, 53.3 mIoU for BEiT on ViT-L).
Table 6 uses classification on iNaturalist and Places. With ViT-H at 448 pixels, MAE reaches 83.4, 86.8 and 88.3 on iNat 2017, 2018, 2019, against previous bests of 75.4, 81.2 and 84.1. On Places205 and Places365 it reaches 66.8 and 60.3, beating results that used 1 billion and 3.5 billion pre-training images.
Read the limits as carefully as the wins. All pre-training uses ImageNet-1K images; the paper does not test more data. All backbones are vanilla ViTs. The target is pixels, which the authors call a lower semantic level than recognition tasks; they note that images are “merely recorded light” with no built-in words, and that random patches do not form meaningful pieces. They hypothesise, but do not prove, that a rich hidden representation explains why the reconstructions still look semantic. The linear-probing score trails contrastive methods. Finally, the broader-impacts note warns that the model predicts content from the statistics of its training set, so it reflects that data’s biases and can generate content that was never there.
Chapter 8
Keep the numbers, the checklist and the core code on one page, and find where to go next
In one paragraph. A masked autoencoder cuts an image into 196 patches, hides a random 75%, and runs a large ViT encoder on the 49 that remain. A small decoder gets those encoded tokens plus one shared mask token for each hidden patch, restores the order, and predicts the pixels of every patch. The loss is mean squared error on the hidden patches only, against per-patch normalised pixels. After training, the decoder is thrown away. The encoder reads whole images and, fine-tuned, reaches 85.9% (ViT-L) and 87.8% (ViT-H at 448 pixels) on ImageNet-1K, while pre-training about 3 times faster than it would with mask tokens in the encoder.
| Item | Value | Where |
|---|---|---|
| Image and patches | 224 × 224, patches 16 × 16: 196 patches of 768 numbers | Sec. 3; Table 3 caption |
| Masking ratio | 75%, so the encoder sees 49 patches | Fig. 5 |
| Encoder (ablations) | ViT-L: 24 blocks, width 1,024 | Sec. 4, Table 1 |
| Decoder (default) | 8 blocks, width 512, 9% of the encoder’s FLOPs per token | Table 1a, 1b |
| Loss | MSE on hidden patches, normalised pixels (85.4 against 84.9 fine-tuning) | Sec. 3, Table 1d |
| Mask tokens in the encoder | Linear probing 59.6% against 73.5% without; 3.3 times more FLOPs | Table 1c |
| Wall-clock (ViT-L, 800 epochs) | 15.4 h against 42.4 h with mask tokens: 2.8 times faster | Table 2 |
| Pre-training recipe | AdamW, base lr 1.5e-4 (lr = base × batch / 256), weight decay 0.05, betas 0.9 and 0.95, batch 4,096, cosine schedule, 40 warmup epochs, random resized crop | Table 8 |
| Fine-tuning recipe | AdamW lr 1e-3, layer-wise lr decay 0.75, batch 1,024, 100 epochs (B) or 50 (L, H), RandAug, mixup 0.8, cutmix 1.0, drop path 0.1 (B, L) or 0.2 (H) | Table 9 |
| Linear probing recipe | LARS lr 0.1, no weight decay, batch 16,384, 90 epochs, a BatchNorm without learned scale in front of the classifier | Table 10 |
| ImageNet-1K, fine-tuned | ViT-B 83.6, ViT-L 85.9, ViT-H 86.9, ViT-H at 448 px 87.8 | Table 3 |
| COCO, ViT-L box AP | 53.3 (supervised 49.3) | Table 4 |
| ADE20K, ViT-L mIoU | 53.6 (supervised 49.9) | Table 5 |
Terms to keep. Patch: a 16 × 16 pixel square. Mask token: one learned vector standing in for a hidden patch, used only in the decoder. Ratio: the share of patches hidden. Linear probing: train only a straight-line classifier on frozen features. Fine-tuning: keep training the whole encoder with labels.
The checklist is the list of decisions you must make to reproduce the paper. Tick each one when you could state the value and the reason without looking back.
Tick what you can explain from memory. The read-out counts them and names the first one left.
A study aid, not a paper result. The eight items match Chapters 1 to 7.
This is the paper’s algorithm in PyTorch-style Python. It follows the “simple implementation” of Section 4: shuffle, cut, encode, append mask tokens, unshuffle, decode, score. Layer definitions are left out; the shapes are the point.
def random_masking(tok, ratio=0.75): # tok: [B, 196, D] B, N, D = tok.shape keep = int(N * (1 - ratio)) # floor(196 * 0.25) = 49 noise = torch.rand(B, N) ids_shuffle = noise.argsort(dim=1) # random order, per image ids_restore = ids_shuffle.argsort(dim=1) # inverse permutation vis = torch.gather(tok, 1, ids_shuffle[:, :keep, None].expand(-1, -1, D)) mask = torch.ones(B, N); mask[:, :keep] = 0 # 1 = hidden, in shuffled order mask = torch.gather(mask, 1, ids_restore) # back to picture order return vis, mask, ids_restore # [B,49,D], [B,196], [B,196] def mae_loss(img): # img: [B, 3, 224, 224] x = patchify(img) # [B, 196, 768] tok = patch_embed(x) + pos_enc # [B, 196, 1024] vis, mask, ids_restore = random_masking(tok) z = encoder(cat([cls_token, vis], 1)) # [B, 50, 1024] no mask tokens z = dec_embed(z) # [B, 50, 512] m = mask_token.expand(B, 196 - 49, -1) # one learned vector, repeated full = cat([z[:, 1:], m], 1) # [B, 196, 512] full = torch.gather(full, 1, ids_restore[:, :, None].expand(-1, -1, 512)) pred = dec_head(decoder(cat([z[:, :1], full], 1) + dec_pos))[:, 1:] # [B, 196, 768] target = x mu, var = target.mean(-1, keepdim=True), target.var(-1, keepdim=True) target = (target - mu) / (var + 1e-6) ** 0.5 # per-patch normalised pixels loss = ((pred - target) ** 2).mean(-1) # [B, 196] per-patch MSE return (loss * mask).sum() / mask.sum() # hidden patches only
MAE sits on three earlier ideas. From language it takes masked modelling: hide, then predict. See the GPT lesson for the other language route, predicting the next word. From vision it takes the ViT of Chapter 1; our attention lesson and positional encoding lesson build the pieces. From earlier self-supervision it takes the autoencoder, and it replaces the contrastive route (contrastive learning and CLIP, self-supervised learning) with something that needs neither negative pairs nor heavy augmentation.
For the multimodal course, the point is the recipe. Hide most of one signal, encode what remains cheaply, and make a light decoder bring back the rest. Lecture 8 shows how the same idea, applied to pieces of images and text, leads to the large multimodal models of the course. Go to Lecture 8.
Now teach it back. Use the Teach and Present buttons at the top to explain the shuffle and the asymmetric design in your own words, or go back to the hero and drag the masking ratio once more. Credit: Masked Autoencoders Are Scalable Vision Learners, by Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár and Ross Girshick (Facebook AI Research). All figures and numbers belong to the authors; the devices, toy scenes and prose here are our own.