What if you threw away convolutions entirely and just treated image patches as tokens? It works -- spectacularly -- if you have enough data.
You have a 224×224 pixel image. You want to classify it -- is it a cat, a car, a sunset? For thirty years, the answer was convolutional neural networks (CNNs). Stacks of small filters slide across the image, learning edges, then textures, then object parts, then whole objects. It works. It won ImageNet. It powers everything.
But transformers have eaten NLP alive. BERT, GPT, T5 -- the same architecture dominates every language task. Transformers have one superpower CNNs lack: global attention. Every token can attend to every other token, no matter how far apart. A CNN needs dozens of layers before a pixel in the top-left can "see" the bottom-right. A transformer does it in one layer.
The problem? A 224×224 image has 50,176 pixels. Self-attention scales quadratically with sequence length. Attending every pixel to every pixel would require 50,176² ≈ 2.5 billion operations per layer. That's absurd.
This is what Dosovitskiy et al. showed in 2020: a pure transformer, applied directly to sequences of image patches, with no convolutional layers at all, matches or beats the best CNNs on image classification. The catch? You need a lot of data. With ImageNet alone (1.3M images), ViT underperforms ResNets. With JFT-300M (300M images), it dominates -- at a fraction of the training cost.
Prior work had tried applying attention to images in limited ways -- local attention windows, sparse attention patterns, attention augmenting CNN feature maps. The radical move of ViT was to say: forget all of that. Take the exact same transformer from NLP. Feed it image patches. See what happens. The answer changed computer vision forever.
Drag the patch size. Watch how dramatically the sequence length drops.
Here is the first step of ViT, and it's beautifully simple. Take your image x ∈ RH×W×C (height H, width W, C color channels). Chop it into a grid of non-overlapping patches, each of size P×P pixels.
For a 224×224 RGB image with P=16:
Each patch is a small 16×16×3 block of pixels -- 768 raw numbers. Flatten it into a single vector: xp ∈ R768. That vector is the "word." The sequence of 196 such vectors is the "sentence." The transformer reads this sentence and classifies the image.
Notice what's missing: no sliding window, no stride, no padding, no learned filters. Just a hard grid. A 5-year-old could do this with scissors. The entire spatial inductive bias of CNNs -- locality, translation equivariance, hierarchical feature extraction -- is thrown away. The transformer must learn all spatial relationships from scratch, using only the data.
There's a subtle but important detail here. The patches are non-overlapping. In a CNN, the first convolutional layer slides a small filter across the image with overlap (stride < kernel size). This means neighboring pixels share computations. ViT's patches have no overlap -- each pixel belongs to exactly one patch. Any information sharing between neighboring patches must happen in the transformer layers, through attention.
The naming convention encodes the patch size into the model name. ViT-B/16 means Base architecture with 16×16 patches (196 tokens from a 224×224 image). ViT-L/32 means Large architecture with 32×32 patches (49 tokens). Smaller patches = more tokens = more compute but finer spatial resolution.
Watch the image get sliced into a grid of patches. Each colored square becomes one token.
Each flattened patch is a 768-dimensional vector (for 16×16×3). But the transformer's hidden dimension D might be different -- 768 for ViT-Base, 1024 for ViT-Large, 1280 for ViT-Huge. We need to map each patch vector into the transformer's embedding space.
This is done with a single trainable linear projection -- a matrix multiply:
For ViT-Base with 16×16 RGB patches: E is a 768×768 matrix. Each raw patch vector (768 numbers of pixel values) gets multiplied by this matrix to produce a 768-dimensional patch embedding.
After projection, each of the N patches is a D-dimensional vector. The image is now a sequence of N embedding vectors, just like a sentence of N word embeddings. From here, the transformer doesn't need to know these came from an image.
nn.Conv2d(3, 768, kernel_size=16, stride=16).The paper inspects the learned projection weights and finds something striking. The top principal components of E look like Gabor filters, oriented edges, and color blobs -- the same features that the first layers of CNNs learn. The transformer discovers these low-level feature extractors purely from data, without any convolutional structure forcing it.
There is also a hybrid variant: instead of raw image patches, feed the output feature maps of a CNN (like ResNet) into the transformer. In this hybrid model, the CNN handles local feature extraction and the transformer handles global reasoning. The paper finds hybrids are slightly better at small compute budgets, but the advantage vanishes at scale. Pure ViT catches up.
After linear projection, the transformer receives 196 patch embeddings. But it has no idea where they came from in the image. Patch 0 could be top-left or bottom-right -- the transformer treats sequences as unordered sets unless you add positional information.
ViT adds learnable 1D position embeddings. One embedding vector per position, learned from scratch during training:
The "+1" is for the CLS token (next chapter). Each position gets its own learned D-dimensional vector, added element-wise to the corresponding patch embedding.
This is a key insight about inductive bias: the 2D structure of images is learnable, not something you need to hard-code. Given enough data, the model figures it out. This finding foreshadows a broader theme in deep learning -- less inductive bias, more data, better results.
There's a practical benefit to 1D position embeddings: resolution flexibility. When fine-tuning at higher resolution than pre-training (say, 384×384 instead of 224×224), the number of patches increases. With 16×16 patches, you go from 196 to 576 tokens. The position embeddings need to be interpolated. Because ViT uses 1D embeddings that have learned 2D structure, you can perform 2D bicubic interpolation on the learned embeddings. The paper finds this works well -- it's the only point where 2D image structure is manually injected.
Click any patch position to see which other positions have similar embeddings. The model learns 2D structure from 1D indices.
We now have 196 patch embeddings with position information. But we need a single vector to classify the image. Which patch should represent the whole image? The top-left? The center? None of them -- each patch only sees a 16×16 region.
ViT borrows BERT's trick: prepend a special learnable [CLS] token to the sequence. This token has no visual content. It starts as a random vector and is trained end-to-end. Its job is to aggregate information from all patches through self-attention and serve as the image-level representation.
After L layers of transformer processing, the CLS token's output z0L has attended to every patch across every layer. It's the image's "summary." A classification head (a small MLP) maps this to class probabilities:
The CLS token is position 0 in the sequence. It gets its own position embedding. During training, it learns to be a "query" that pulls information from the patch tokens. During inference, it's the only token whose output matters for classification.
During pre-training (on JFT-300M or ImageNet-21k), the classification head is a small MLP with one hidden layer (hidden dim = 3072 for ViT-Base, GELU activation, tanh on the output). During fine-tuning on a downstream task, the pre-trained head is discarded and replaced with a single linear layer: (768,) → (K,) mapping from D dimensions to K classes. This is the only part of the model that changes shape between tasks.
When fine-tuning at higher resolution (e.g., 512×512 instead of 224×224), the number of patches increases but the CLS token stays the same. It's just one more token in the sequence. The transformer handles variable-length sequences natively -- the only adjustment needed is interpolating the position embeddings, not restructuring the model.
From here, ViT uses a completely standard transformer encoder. No modifications. No vision-specific layers. The same architecture that processes sentences now processes image patch sequences.
Each layer has two sub-blocks with pre-norm residual connections:
Multi-head self-attention (MSA) lets every patch attend to every other patch. The CLS token attends to all patches. Patch 42 in the bottom-right can attend to patch 3 in the top-left -- global context in one layer. This is what CNNs need dozens of layers to achieve.
The MLP block has two linear layers with GELU activation and a hidden dimension 4× the embedding dimension (e.g., 3072 for ViT-Base with D=768). The MLP processes each token independently -- it's the attention that mixes information across patches.
Note the pre-norm design: LayerNorm is applied before each sub-block, not after. This differs from the original Vaswani et al. transformer which uses post-norm. Pre-norm makes training more stable, especially at large scale. It's become the default in modern transformers.
One important detail: the self-attention layers are the only place information flows between patches. In a CNN, local information flows implicitly through overlapping receptive fields that gradually expand layer by layer. In ViT, every patch can talk to every other patch in every layer. This global connectivity is both ViT's superpower (long-range reasoning in one layer) and its weakness (no built-in locality bias, so it needs massive data to learn what's local and what's distant).
| Model | Layers | Hidden D | MLP size | Heads | Params |
|---|---|---|---|---|---|
| ViT-Base | 12 | 768 | 3072 | 12 | 86M |
| ViT-Large | 24 | 1024 | 4096 | 16 | 307M |
| ViT-Huge | 32 | 1280 | 5120 | 16 | 632M |
Here is the twist that makes ViT interesting beyond just "transformers work on images." When trained on ImageNet alone (1.3M images), ViT-Large underperforms ViT-Base. The bigger model is worse. Why?
CNNs have strong inductive biases baked into their architecture: locality (each filter sees only a small patch), translation equivariance (the same filter detects a cat ear whether it's top-left or bottom-right), and hierarchical structure (low-level features compose into high-level ones). These biases act as a form of implicit regularization -- they tell the model how images work before it sees any data.
ViT has almost none of this. The only image-specific bias is the initial patch extraction. Everything else -- spatial relationships, locality, hierarchy -- must be learned from data. With limited data, the model overfits. It memorizes rather than generalizes.
The paper also shows a remarkable efficiency result. ViT-H/14 pre-trained on JFT-300M reaches 88.55% ImageNet accuracy using 2,500 TPUv3-core-days. The comparable CNN (BiT-L, ResNet-152x4) uses 9,900 TPUv3-core-days for 87.54%. ViT is 4× more compute-efficient at the frontier -- it just needs the data to get there.
The scaling study reveals a clean pattern: at the same compute budget, ViT consistently outperforms ResNets on transfer learning. The gap widens as compute increases. And unlike ResNets, ViT shows no sign of saturation -- performance keeps improving with more compute, suggesting even larger models would do even better. This observation anticipated the scaling law era.
For practitioners, the takeaway is clear. If you have a small dataset (< 1M images), use a CNN or a ViT pre-trained on a large dataset and fine-tuned. If you have hundreds of millions of images and the compute to match, ViT will give you better results per FLOP.
Drag the dataset size. On small data, CNN wins. On large data, ViT pulls ahead.
This is the full ViT pipeline in action. Below, you'll see an image split into 16×16 patches. Each patch is projected into an embedding, given a position, and passed through a single self-attention layer. Click any patch to see which other patches attend to it most strongly.
The attention patterns reveal what the model has learned. Early layers tend to show local attention -- nearby patches attend to each other, mimicking the locality of convolutions. Deeper layers show global patterns -- semantically related patches attend to each other regardless of distance. The CLS token (position 0) gradually learns to attend to the most informative patches.
Click any patch to see its attention pattern. Brighter = stronger attention. Toggle layers to see how attention evolves.
In the real ViT, this attention operation happens 12 times (for ViT-Base) with 12 heads per layer. Each head learns different attention patterns -- one head might focus on texture similarity, another on spatial proximity, a third on color. The heads' outputs are concatenated and projected, then the MLP processes each token independently before the next layer.
The paper analyzed real attention patterns and found a striking progression. In early layers, many attention heads show local behavior -- they attend primarily to nearby patches, effectively mimicking the local receptive fields of a CNN. In deeper layers, attention becomes global -- heads attend to semantically related patches regardless of distance. Some heads even specialize: the paper found heads that attend primarily along rows or columns, heads that attend to patches of similar color, and heads that attend to edges.
This discovery has a profound implication. The hierarchical processing of visual information -- local edges composing into textures, textures into parts, parts into objects -- is not an artifact of the CNN architecture. It's a property of visual data itself. Give a sufficiently powerful model enough data, and it will rediscover this hierarchy on its own.
ViT is a milestone paper, but it's also a starting point. Its deliberate simplicity -- standard transformer, no vision-specific tricks -- proved that the transformer architecture is universal. Everything that followed built on this foundation.
| Method | Key Idea | Relation to ViT |
|---|---|---|
| DeiT (2021) | Knowledge distillation from CNNs | Makes ViT work with less data |
| Swin Transformer (2021) | Shifted window attention | Adds back locality for efficiency |
| MAE (2022) | Masked autoencoder pre-training | Self-supervised ViT pre-training |
| CLIP (2021) | Vision-language contrastive learning | Uses ViT as the image encoder |
| DINO (2021) | Self-distillation with no labels | ViT learns object segmentation without supervision |
The deepest lesson of ViT is about the relationship between inductive bias and data. CNNs encode strong assumptions about images -- locality, translation equivariance, hierarchy. These assumptions help when data is scarce, but limit the model when data is abundant. ViT makes almost no assumptions, leaving the model free to discover whatever structure exists in the data. This requires more data but ultimately leads to better representations.
The paper also briefly explored self-supervised pre-training, predicting masked patches (like BERT's masked language modeling). The self-supervised ViT-B/16 reached 79.9% on ImageNet -- promising but below the supervised version. This planted the seed for MAE (He et al., 2022), which fully realized this idea and achieved state-of-the-art results with masked image modeling.
Perhaps most remarkably, DINO (Caron et al., 2021) showed that ViT trained with self-distillation (no labels at all) learns features that naturally segment objects -- the attention maps in the last layer cleanly separate foreground from background. No CNN had ever shown this emergent behavior. Something about the global attention mechanism, combined with sufficient scale, enables ViT to discover object boundaries as a side effect of representation learning.
python # The complete ViT forward pass in pseudocode def vit_forward(image, patch_size=16): # 1. Split into patches: (224,224,3) -> (196, 768) patches = split_into_patches(image, patch_size) # 2. Linear projection: (196, 768) -> (196, D) embeddings = patches @ E # E is (768, D) # 3. Prepend CLS token: (196, D) -> (197, D) embeddings = concat([cls_token, embeddings]) # 4. Add position embeddings: (197, D) + (197, D) embeddings = embeddings + pos_embeddings # 5. Transformer encoder: L layers for layer in transformer_layers: embeddings = layer(embeddings) # 6. Classification: take CLS output -> class probs return mlp_head(layer_norm(embeddings[0]))
That's it. The entire ViT architecture fits in 15 lines of pseudocode. There is no pooling, no batch normalization, no convolutional layer, no multi-scale feature pyramid. Just patches, a linear projection, positional embeddings, and a standard transformer. The simplicity is the point.