The paper that mapped the scaling laws of vision: how error falls as a power law in compute, data, and model size — and built ViT-G/14, a two-billion-parameter Transformer reaching ~90.45% ImageNet top-1 by riding that curve to its edge.
It is 2021. You run a vision team. You have a Vision Transformer that scores well on ImageNet, and your manager asks a deceptively simple question: "If I gave you ten times the GPUs, how much better would the model get?"
You do not know. Nobody does. You could train a bigger model and find out — but a single large run costs weeks of TPU time and tens of thousands of dollars. You cannot afford to guess. You need a law: a formula that, given a compute budget, predicts the accuracy you will get — before you spend a cent on the run.
Language modeling already had such laws. Kaplan et al. (2020) had shown that for autoregressive language models, the test loss falls as a smooth power law in model size, dataset size, and compute. The curve was so clean you could extrapolate across orders of magnitude. But vision was an open question. Convolutional networks (ResNets, EfficientNets) had famously diminishing returns — pile on parameters and accuracy plateaus or even degrades. Was the Transformer different? Did vision obey a scaling law at all?
The fear (left curve, teal): accuracy plateaus — doubling compute buys almost nothing past a point. The hope (right curve, warm): error keeps falling as a straight line on a log-log plot, so each 10× of compute buys a predictable, constant drop. Drag to sweep the compute budget and watch which world you would rather live in.
Zhai, Kolesnikov, Houlsby, and Beyer set out to settle it. They trained Vision Transformers across five orders of magnitude of compute — from tiny models on small data to a two-billion-parameter giant on a three-billion-image dataset (an internal Google set called JFT-3B) — and measured the relationship between compute and error. The headline finding: vision does obey a scaling law, and it is remarkably clean. But there is a twist that language did not show as sharply — a saturating tail, a floor the error approaches but cannot cross with compute alone.
Throughout, compute (C) is the total training cost — roughly the number of images seen multiplied by the cost-per-image, which itself grows with model size. The paper plots it in "TPUv3-core-days," but for our purposes think of C as a single number on a log axis spanning from ~1 (a small ViT-Ti on a little data) to ~10,000+ (ViT-G on JFT-3B). Error (E) is the few-shot or fine-tuned top-1 error rate — one minus accuracy — because error is what falls as a power law, not accuracy.
The rest of this lesson rebuilds their result from the ground up: why a straight line on a log-log plot is a power law (Ch 1), why vision needs a saturating floor (Ch 2), how the three knobs — model size, data, compute — interact (Ch 3), what "compute-optimal" means and why a bigger model is not always better (Ch 4), the engineering tricks that made a 2B-parameter ViT fit in memory (Ch 5), the recipe for ViT-G itself (Ch 6), the few-shot results (Ch 7), and a live frontier explorer (Ch 8).
The whole paper lives on one kind of plot: error versus compute, both axes logarithmic. On that plot the data falls on a near-straight line. Before we trust anything else, we need to understand why a straight line on log-log axes means the underlying relationship is a power law — and how to read its slope.
A power law is any relationship of the form:
Here E is error, C is compute, a is a scale constant (where the line sits vertically), and b > 0 is the scaling exponent (how steeply error falls). Take the logarithm of both sides:
That is the equation of a straight line in the variables (log C, log E): intercept log a, slope −b. So if you plot log E against log C and see a line, the slope is the negative exponent. A steeper line means a larger b means error falls faster per decade of compute.
Suppose at compute C₁ = 100 you measure error E₁ = 0.20, and at C₂ = 10,000 (two decades later) you measure E₂ = 0.10. The exponent is the slope of the log-log line through these two points:
Plug in base-10 logs. The numerator: log(0.10) − log(0.20) = −1.000 − (−0.699) = −0.301. The denominator: log(10000) − log(100) = 4.000 − 2.000 = 2.000. So:
So b ≈ 0.15. Sanity check with the multiplier rule: per decade, error should fall by 10−0.15 ≈ 0.708. Two decades: 0.7082 ≈ 0.501 — and indeed 0.20 × 0.501 ≈ 0.10. The arithmetic is self-consistent. Now recover a from either point: a = E₁ · C₁b = 0.20 × 1000.15 = 0.20 × 1.995 ≈ 0.399.
That is the entire machinery of fitting a (pure) power law: take logs, fit a line, read off slope and intercept, exponentiate back. The Code Lab at the end of Chapter 4 makes you do exactly this on synthetic ViT data.
The identical power law E = a·C−b drawn on linear axes (left — a vague drooping curve) and on log-log axes (right — a clean straight line whose slope is −b). Drag the exponent and watch the linear curve bend while the log-log line just tilts. This is why everyone in scaling research lives in log-log space.
If a pure power law E = a·C−b held forever, then enough compute would drive error to zero: as C → ∞, C−b → 0. That is obviously false. Some images are genuinely ambiguous; labels are noisy; the test set has irreducible error. Real systems approach a floor, not zero. Zhai et al. capture this with a saturating power law:
The single extra term, the constant c, changes everything. Now c is the irreducible error — the asymptote the curve bends toward but never crosses. The first term a·C−b is the reducible error, the part compute can still buy back. As C grows, the reducible part shrinks toward 0 and E settles onto c.
Take logs of the saturating law and you no longer get a straight line:
At small C, the a·C−b term dominates and the curve looks like the pure power law (a near-straight log-log line of slope −b). At large C, the c term dominates and log E flattens to the constant log c — the line goes horizontal. The transition happens where the two terms are equal: a·C−b = c, i.e. at the knee compute
Below the knee, buying compute is great. Above the knee, you are scraping diminishing reducible error against a fixed floor and each decade buys far less.
Take a = 0.40, b = 0.15, c = 0.08 (numbers in the spirit of the paper's few-shot fits — a floor around 8% error). First the knee:
Compute 56.667: log₁₀(5) = 0.699, times 6.667 = 4.66, so Cknee = 104.66 ≈ 45,700. Now compare two budgets around the knee. At C = 5,000 (well below): E = 0.40×5000−0.15 + 0.08. Compute 5000−0.15: log(5000)=3.699, times −0.15 = −0.555, so 10−0.555 = 0.279; reducible = 0.40×0.279 = 0.112; E ≈ 0.192. At C = 50,000 (10×, near the knee): 50000−0.15: log = 4.699, times −0.15 = −0.705, 10−0.705 = 0.197; reducible = 0.079; E ≈ 0.159.
So a 10× compute spend bought us from 19.2% to 15.9% error — a 3.3-point drop. Now spend another 10×, to C = 500,000 (above the knee): 500000−0.15: log = 5.699, times −0.15 = −0.855, 10−0.855 = 0.140; reducible = 0.056; E ≈ 0.136. The second decade bought only 2.3 points, the third would buy less. The floor c = 0.08 is pulling the curve flat. That is the saturating tail, and it is exactly the regime ViT-G operates in.
Error vs. compute on log-log axes for E = a·C−b + c. The teal dashed line is the irreducible floor c; the warm curve is the full law. Raise c and the curve bends earlier and harder — past the knee, compute stops paying. Drag c and find the knee marker move.
So far we have treated "compute" as a single knob. But compute is really the product of two others: how big the model is, and how much data you push through it. Zhai et al.'s central experimental design was a 2D sweep over model size and data size, with compute as the diagonal you can spend along. The three knobs interact, and the interactions are the whole story.
| Knob | Symbol | What it controls | Failure mode if mismatched |
|---|---|---|---|
| Model size | N (params) | Representational capacity — what the model can learn | Too big for the data → overfits / wastes compute |
| Data size | D (images seen) | How much signal there is to learn from | Too much for the model → model bottlenecks, data wasted |
| Compute | C ≈ N · D | Total training cost (the budget you actually pay) | Spent on the wrong N:D split → below the frontier |
The key empirical findings, stated plainly:
Finding 1 — Bigger models are more sample-efficient. A larger ViT reaches a given error with fewer images seen than a smaller one. Capacity lets the model extract more from each example. So if you are data-rich, scale the model.
Finding 2 — Big models need big data. A two-billion-parameter ViT trained on a small dataset overfits and underperforms a smaller model. Capacity is only useful if there is enough signal to fill it. JFT-3B (≈3 billion images) existed precisely so ViT-G would not starve. On ImageNet-1k alone, ViT-G would be a disaster.
Finding 3 — There is a compute-optimal split. For any compute budget C, there is a best way to divide it between model size and data. Spend too much on a giant model and too little data, you overfit; spend it all on data with a tiny model, you bottleneck on capacity. The optimum traces a curve in (N, D) space — the compute-optimal frontier, the subject of Chapter 4.
Say your budget is C = N·D = 1,000 units (in arbitrary model-units × data-units). Compare splitting it as a big model, little data run (N = 100, D = 10) versus a balanced run (N = 10, D = 100). Suppose error follows a toy two-factor law where each factor saturates: E = 0.5·N−0.15 + 0.5·D−0.15 + 0.05 (capacity term + data term + floor).
Big/little: N−0.15 = 100−0.15: log(100)=2, ×−0.15=−0.30, 10−0.30=0.501; term = 0.250. D−0.15 = 10−0.15 = 0.708; term = 0.354. E = 0.250 + 0.354 + 0.05 = 0.654.
Balanced: N−0.15 = 10−0.15 = 0.708; term = 0.354. D−0.15 = 100−0.15 = 0.501; term = 0.250. E = 0.354 + 0.250 + 0.05 = 0.654.
Identical here only because the toy law is symmetric. Make the data term steeper (real data is often the binding constraint) — say the data exponent is 0.20 instead of 0.15 — and the balanced run that gives data more room wins. The lesson: the split matters, and the right split depends on which factor has the steeper, less-saturated curve in your regime. The interactive below lets you feel this.
Fixed budget C = N·D. Slide the split between model size (warm) and data size (teal). The bar shows resulting error from a toy two-factor saturating law; the marker flags the optimum. Find the split that minimizes error — and notice it is rarely "all model" or "all data."
Now we make "spend compute wisely" precise. Imagine a whole family of training runs, each a point in (compute, error) space. Small models plateau early (they hit their capacity floor); large models start expensive but keep improving. Connect the best achievable error at each compute level and you get the compute-optimal frontier — a lower envelope. Every real run sits on or above it. The frontier is the promise: "with this budget, spent optimally, here is the error you can reach."
At a tiny budget, a giant model cannot even finish a useful amount of training — its per-step cost eats the whole budget before it learns much, so a small model that completes many steps wins. At a huge budget, a small model has long since saturated (hit its capacity floor) and is wasting every extra image, while a giant model still has reducible error to spend on. So as C rises, the crossover point moves and the best N rises with it. This is the vision version of the language result that bigger budgets justify bigger models — and the seed of the compute-optimal model-sizing recipes that followed.
The procedure the paper effectively uses, and that you will replicate in the Code Lab: (1) train many models of different sizes for varying durations; (2) plot all the (C, E) points; (3) for each compute level take the minimum error; (4) fit the saturating law E = a·C−b + c to that lower envelope; (5) extrapolate to a budget you have not run yet to predict its error — and to choose the model size you should train there. ViT-G was, in effect, a point chosen by extrapolating the frontier: the fit said there was reducible error left, so a bigger model at a bigger budget should pay off. It did.
Four model sizes (Ti, S, B, L — cool to warm), each its own saturating E-vs-C curve on log-log axes. The thick warm line is their lower envelope — the compute-optimal frontier. Drag the budget marker and read off which model size is optimal there: small at low compute, large at high compute. The optimal model grows with the budget.
Suppose the frontier fit on your smaller runs gives a = 0.42, b = 0.13, c = 0.075 (compute in TPU-core-days). You have spent up to C = 2,000 and reached E = 0.42×2000−0.13 + 0.075. Compute 2000−0.13: log(2000)=3.301, ×−0.13=−0.429, 10−0.429=0.372; reducible = 0.156; E ≈ 0.231 (23.1% error). Your manager offers 25× more compute (C = 50,000). Predict: 50000−0.13: log=4.699, ×−0.13=−0.611, 10−0.611=0.245; reducible=0.103; E ≈ 0.178 (17.8%). That is a 5.3-point predicted gain — and a reducible term (0.103) still well above zero, meaning the floor is not yet reached. That calculation — done before spending a TPU-hour — is what justifies committing to the big run. This is exactly the kind of reasoning the lab below makes concrete.
A scaling law is only useful if you can actually build the model at the top of the curve. ViT-G/14 has on the order of two billion parameters. Naively, that does not fit on accelerators alongside optimizer state and activations. The paper's second contribution is a bag of engineering tricks that shrink the memory footprint enough to train the giant — without which the whole frontier is theoretical.
The classifier head (the final linear layer mapping the representation to class logits) behaves differently from the body. The authors found that applying a separate, stronger weight decay to the head — decoupled from the body's weight decay — improved few-shot transfer. Intuitively, a heavily-regularized head forces the body to learn a more general, transferable representation rather than letting the head memorize. It is a one-line change with an outsized effect on downstream few-shot accuracy.
The original ViT prepends a learned [class] token whose final representation is fed to the head. The scaling paper instead uses global average pooling (GAP) over all patch tokens — average the final-layer patch embeddings into one vector — and a small change to the head normalization. This removes a special-cased token, simplifies the architecture, and works at least as well at scale. One fewer thing to scale, one fewer thing to break.
The dominant memory cost at this scale is optimizer state. Adam keeps two extra full-size tensors per parameter (first and second moments) — so 2B params means ~6B numbers in optimizer state alone in fp32. The paper adapts Adafactor: instead of storing the full second-moment matrix for each weight matrix, it stores only per-row and per-column statistics (a rank-one factorization), slashing the second-moment memory from O(mn) to O(m+n) per weight matrix. This is what actually let ViT-G fit and train.
Take N = 2×109 parameters in fp32 (4 bytes each). The weights themselves: 2e9 × 4 = 8 GB. Adam adds two more buffers of the same size: first moment 8 GB + second moment 8 GB = 16 GB of optimizer state, for 24 GB total just for weights + optimizer. Adafactor keeps the first moment (8 GB) but factorizes the second moment to a small fraction — call the factored second-moment state negligible by comparison (kilobytes-to-megabytes per matrix). So the optimizer-state bill drops from ~16 GB toward ~8 GB. On accelerators with tens of GB, freeing 8 GB is the difference between "fits" and "out of memory" — and that is before counting activations, which gradient checkpointing further tames.
| Component | Original ViT | Scaling-ViT change | Why |
|---|---|---|---|
| Pooling / head input | [class] token | Global average pooling | Simpler, scales cleanly, no special token |
| Head regularization | Shared weight decay | Decoupled (stronger) head decay | Forces transferable body representation |
| Optimizer | Adam (full 2nd moment) | Adafactor (factored 2nd moment) | O(mn) → O(m+n) optimizer memory |
| Precision / activations | fp32 everywhere | Half-precision + checkpointing | Cuts activation + state memory |
For a square weight matrix of side d, Adam's second-moment buffer needs d² numbers (warm); Adafactor's factored row+col statistics need 2d numbers (teal). Slide d up and watch the gap explode — at d = 4096 the bars are off by orders of magnitude. This factorization is what fit ViT-G in memory.
Time to assemble the trophy. ViT-G/14 ("G" for giant, "14" for a 14×14 patch size) is the largest model in the study, on the order of two billion parameters. Let us trace the data flow with concrete tensor shapes so you could, in principle, build it.
An input image, say 224×224×3, is cut into non-overlapping patches of 14×14 pixels. The number of patches is (224/14)×(224/14) = 16×16 = 256 patches. Each patch (14×14×3 = 588 numbers) is flattened and linearly projected to the model width d. So the input becomes a sequence of 256 tokens, each a d-vector: shape [256, d]. At higher fine-tuning resolutions (e.g. 518×518) the patch count grows (here 37×37 = 1369), which is one lever for squeezing out extra accuracy.
A learned positional embedding is added to each of the 256 tokens (so the model knows patch order). Unlike the original ViT, no [class] token is prepended — we will pool at the end instead. Shape stays [256, d].
The 256 tokens pass through L identical Transformer encoder blocks. Each block is the standard recipe: multi-head self-attention (all 256 patches attend to all 256), Add & LayerNorm, an MLP (feed-forward) of hidden width ~4d, Add & LayerNorm. For ViT-G, width d, depth L, and head count are all pushed large (the paper's largest configuration). Shape through the stack: [256, d] in, [256, d] out.
After the final block, average the 256 patch tokens into a single d-vector (GAP): [256, d] → [d]. A final LayerNorm and a linear head map [d] to the number of classes. During pretraining on JFT-3B the head predicts JFT's label set; for ImageNet evaluation the head is re-fit (fine-tuned) or used few-shot. Shape: [d] → [num_classes].
Walk the tensor through the model. Click "Next Stage" to advance: image → 256 patches → +pos → L encoder blocks → GAP → head. The highlighted box shows the active stage and its shape. This is the whole forward pass of a giant ViT in five moves.
The bulk of a ViT's parameters are in the per-block matrices. Each encoder block has: attention projections ≈ 4d2 (Q, K, V, and output, each d×d), plus an MLP of 2·d·(4d) = 8d2 (up-project d→4d, down-project 4d→d). So per block ≈ 12d2. With L blocks: ≈ 12·L·d2. To land near 2×109 params: solve 12·L·d2 = 2e9. If L = 48, then d2 = 2e9/(12×48) = 3.47e6, so d ≈ 1863 — a width near ~1600–1900 with ~48 layers gets you to billions. (The published ViG-G config is in this neighborhood: very wide, ~48 layers, MLP ~4×.) The point is that d2 scaling means width dominates the parameter count — doubling width quadruples params per block.
How do you measure the quality of a representation cheaply, across hundreds of training runs, without a full fine-tune each time? The paper leans on few-shot evaluation — and the choice of metric is itself part of the methodology.
For 10-shot ImageNet, you freeze the pretrained model, take just 10 labeled images per class, compute their representations, and fit a simple linear classifier (often closed-form, like a regularized least-squares / "linear probe") on top. No backprop through the giant model, no long fine-tune — a few minutes of CPU/GPU. This makes it cheap enough to evaluate every point on the scaling sweep, which is exactly what you need to draw a frontier. The few-shot accuracy tracks the full fine-tuned accuracy well enough to rank models.
Riding the frontier to its measured edge, ViT-G/14 reaches about 90.45% top-1 accuracy on ImageNet (with fine-tuning at high resolution) — a state-of-the-art result at publication, crossing the 90% line. On few-shot, the larger models dominate: ViT-G's frozen representation gives strong 10-shot ImageNet accuracy, far above smaller ViTs, confirming the scaling trend holds for transfer, not just in-distribution accuracy. Critically, the (compute, error) points across the whole sweep fit the saturating power law cleanly — which is the scientific result, with the 90.45% being its most quotable consequence.
| Quantity | Value / finding | Why it matters |
|---|---|---|
| ViT-G/14 ImageNet top-1 | ~90.45% | First ViT past 90%; trophy at the frontier's edge |
| Scaling fit | E ≈ a·C−b + c, clean across ~5 orders of compute | The law is real and predictive for vision |
| Few-shot trend | Bigger ViTs → better 10-shot transfer | Scaling improves representations, not just fit |
| Data requirement | JFT-3B (≈3B images) to feed ViT-G | Model+data must scale together |
| Reducible error at ViT-L | Still positive (floor not reached) | Justified spending on the bigger ViT-G |
Say ViT-B at compute C = 3,000 gives 10-shot ImageNet accuracy 78.0%, so error E = 1 − 0.780 = 0.220. ViT-L at C = 30,000 gives 84.5%, so E = 0.155. Those two (C, E) points already let you estimate a local slope (Chapter 1 machinery): b = −[log(0.155)−log(0.220)]/[log(30000)−log(3000)] = −[−0.810−(−0.658)]/[4.477−3.477] = −(−0.152)/1.000 = 0.152. A local b ≈ 0.15 — consistent with the global fit, and exactly the kind of cheap measurement few-shot enables at every scale. String hundreds of these together and the frontier emerges.
Simulated 10-shot ImageNet top-1 accuracy (warm) vs. model scale, with the saturating ceiling drawn in teal. Slide the model scale from Ti to G and watch accuracy rise toward — but never reach — the ceiling (the 1 − c floor). The last few points are ViT-G territory, deep in the saturating tail.
This is the payoff — the paper's whole apparatus in one live instrument. You control the three fit parameters of the saturating law and a compute budget. The plot draws the law on log-log axes, marks the floor c, the knee, your chosen budget, and reports the predicted error, the per-decade return at your operating point, and where ViT-G/14 would land. Push the parameters and feel why the authors chose to build a giant.
Drag a (scale — vertical height above floor), b (exponent — descent steepness), c (irreducible floor), and the budget. The warm curve is E = a·C−b + c on log-log axes; the teal dashed line is the floor c; the dotted vertical is the knee (where reducible = floor); the marker is your budget. The readout gives predicted error and the marginal return of the next 10×.
Things to try. (1) Crank c up: the curve flattens and the knee slides left — past it, even huge budgets barely move E. That is a benchmark or dataset whose floor you have already hit; building a bigger model would be wasteful. (2) Drop c toward 0 and raise b: now every decade of compute pays handsomely and the line stays steep — the regime where scaling is a license to print accuracy. (3) Use the presets: "ViT-L regime" sits before the knee with reducible error left; "Jump to ViT-G" pushes the budget far right and shows the predicted ~90%-accuracy point deep in the tail. The gap between those two presets — the reducible error ViT-L left on the table — is exactly the headroom that made ViT-G worth its TPUs.
The scaling law is powerful but it is a description, not a guarantee. Here are its honest limits, where it connects, and a one-page summary.
Extrapolation is risky. A fit on five orders of compute predicts well near the data, but the saturating form could mis-estimate c, and a new regime (a different architecture, a data-quality cliff, a distribution shift) can break the curve. The law is empirical — it has no first-principles derivation guaranteeing it holds at 100× the largest measured budget.
The floor is partly the benchmark. Much of c on ImageNet is label noise and ambiguity, not model limitation. So "90.45%" measures the dataset as much as the model, and the law's asymptote is dataset-specific.
Compute is not the only axis. Data quality, augmentation, resolution, and the pretraining objective all shift the curve. JFT-3B is a particular (proprietary) dataset; the exact a/b/c are not universal constants.
It says nothing about robustness, calibration, or fairness. A model can ride the accuracy frontier and still be brittle out-of-distribution. The law optimizes one number (top-1 error); the things it ignores are where vision research went next.
This paper, alongside language scaling laws, set up the modern playbook: measure a small sweep, fit a law, extrapolate to choose the big run. The following year, compute-optimal model/data balancing (Chinchilla, in language) made the "balance all three knobs" lesson quantitative. In vision, scaling-law thinking fed SigLIP, ViT-22B, and the data-and-compute-matched training recipes behind frontier multimodal models. The instinct "do not just make it bigger — ride the frontier" is now standard.
| Concept | Formula / fact | One-line meaning |
|---|---|---|
| Pure power law | E = a·C−b | Straight line on log-log; slope = −b |
| Saturating law | E = a·C−b + c | Power law that bottoms out at floor c |
| Exponent b | 10× compute → E × 10−b | Return on each decade of compute |
| Floor c | limC→∞ E = c | Irreducible error (data/label noise) |
| Knee | Cknee = (a/c)1/b | Where reducible error = floor; returns flatten after |
| Frontier | min over models at each C | Lower envelope; optimal model grows with budget |
| Three knobs | C ≈ N · D | Scale model + data + compute together |
| Memory trick | Adafactor: O(mn) → O(m+n) | Factored 2nd moment fits the 2B-param giant |
| Head trick | GAP + decoupled head decay | Simpler pooling, more transferable body |
| Trophy | ViT-G/14 ≈ 90.45% ImageNet | First ViT past 90%, at the frontier's edge |
To build the prerequisites and follow the thread:
"What I cannot create, I do not understand." — Feynman. A scaling law lets you understand a model you have not yet created — and decide whether to.