Scaling · Vision & Vision-Language Models

Vision & VLM Scaling LawsHow Accuracy Buys Parameters, Data & Compute

You have a budget and a question no architecture diagram can answer: how big should the model be? Double the parameters? Double the data? Both? Scaling laws turn that guess into arithmetic. They are the empirical physics of deep learning — messy power laws on log-log paper that say, with eerie regularity, exactly how much accuracy your next dollar of compute will buy, and exactly where the curve stops paying out.

Prerequisites: you can read a graph, you remember what a logarithm does (turns multiplication into addition), and you've met the words parameters and training data. If "a bigger model trained on more images tends to be more accurate" sounds obviously true, you're ready to find out how true, and where it breaks.
10
Chapters
9
Simulations
0
Assumed Knowledge

Chapter 0: The Million-Dollar Question

You are standing at a whiteboard with a research budget. Your team wants to train an image model that beats your current one. Someone says "make it bigger." Someone else says "no, we need more data." A third person, who has signed the cloud invoices, asks the only question that actually matters: if I have one fixed pile of money, how do I spend it to get the most accuracy? Bigger model? More images? A longer training run? Some mix? That single allocation decision can swing your final accuracy by ten points and your bill by a factor of ten.

Here is why guessing is dangerous. Training a frontier vision model is not cheap curiosity. A run of the Scaling Vision Transformers work — the study that produced the two-billion-parameter ViT-G — consumed thousands of accelerator-days and a dataset of three billion images. At cloud rates that is six or seven figures of compute per serious run. You do not get to try every combination of model size, dataset size, and training length. You get a handful of expensive shots, and you must aim them with something better than vibes.

The naive instinct is "spend it all on parameters — the biggest model wins." Watch what happens when we test that instinct. In the widget below you have a fixed compute budget (think: a fixed area you can fill). You slide that budget between model size and training data. Push it all the way to "huge model, tiny data" and the model starves: a giant network shown too few images memorizes and generalizes poorly, so error stays high. Push it all the way to "tiny model, mountains of data" and the model saturates: a small network simply lacks the capacity to absorb what the data is offering. The lowest error sits at neither extreme — it sits at a balanced split that the curve will tell you exactly how to find.

Spend a fixed budget: all parameters? all data? or balanced?

A fixed compute budget, split between model size and data. Drag the split and watch test error. Both extremes are bad — a starved giant or a saturated dwarf. The minimum lives in the middle, and finding it is what this whole lesson is about.

Split: data ←→ params 0.50
The misconception to kill on day one: "the biggest model always wins, so just scale parameters." False, and expensively so. A model's accuracy is bounded by three resources at once — parameters, data, and compute — and starving any one of them caps the other two. A 2-billion-parameter model trained on 10,000 images is worse than a 100-million-parameter model trained on 100 million images. Scaling is an allocation problem, not a "make number big" problem. The rest of this lesson is the math that does the allocation for you.

The astonishing thing — the reason this is a science and not just folklore — is that the relationship between resources and error is not chaotic. It is a power law: a clean, predictable mathematical shape that, when you plot it the right way (on log-log axes), turns into a straight line. Straight lines are wonderful. You can fit them from a few cheap small runs and extrapolate to predict the accuracy of a run you can't afford to try. That is the superpower scaling laws give you: forecasting the result of a million-dollar experiment from a thousand-dollar pilot.

Over the next nine chapters we will earn that superpower from scratch. We will derive what a power law is and read it on log-log paper (Chapter 1), watch the famous saturating ViT-G curve and understand why it bends (Chapter 2), solve the budget-allocation problem like the Chinchilla paper did (Chapter 3), see why CLIP and LAION made data quality the new frontier (Chapter 4), learn how zero-shot and linear-probe transfer scale (Chapter 5), confront the irreducible label-noise ceiling that no amount of money can buy past (Chapter 6), then build a tool that fits a scaling law to noisy data and extrapolates with error bars (the showcase, Chapter 7), and finish with the engineer's decision tree — scale, distill, probe, or fine-tune (Chapter 8) — and the map of where this connects (Chapter 9).

You have a fixed compute budget. You pour all of it into a gigantic model and train it on only a tiny dataset. What is the most likely outcome, and why?

Chapter 1: Power Laws — Straight Lines in Disguise

Everything in scaling rests on one mathematical shape, so let us build it carefully. A power law says one quantity is a constant times another raised to a fixed exponent. For our purposes: error falls as a resource grows, like error = A · N−α, where N is the amount of a resource (parameters, or images, or compute), A is a scale constant, and α (alpha) is the scaling exponent — the single number that says how fast error drops as you add resource. The minus sign just encodes "more resource, less error."

Why does this specific shape keep showing up across vision, language, and audio models? The honest answer is: it is an empirical observation, not a theorem handed down from first principles. People trained models at many scales, plotted the results, and a power law fit astonishingly well over many orders of magnitude. There are partial theories (about how a model carves up a data distribution into ever-finer pieces), but you should hold the power law the way physicists held Kepler's laws before Newton: a reliable, predictive regularity whose deep "why" is still debated. What matters is that it works and lets you forecast.

The magic move is the log-log plot. Take logs of both sides of error = A · N−α. Using the rule that the log of a product is a sum, and the log of a power pulls the exponent out front, we get log(error) = log(A) − α · log(N). Stare at that. If you let y = log(error) and x = log(N), it reads y = log(A) − α·x — the equation of a straight line with slope −α and intercept log(A). A messy curve on ordinary axes becomes a ruler-straight line on log-log axes. That is why every scaling-laws plot you will ever see uses log scales on both axes: it makes the law visible and the exponent readable as a slope.

Reading a slope as an exponent, by hand. Suppose on a log-log plot you have two points. At N = 1 million parameters the error is 0.40; at N = 100 million it is 0.20. The error halved while parameters grew 100×. The slope on log-log axes is (change in log error) / (change in log N). Using base-10 logs: log(0.20) − log(0.40) = −0.699 − (−0.398) = −0.301. And log(100M) − log(1M) = 8 − 6 = 2. Slope = −0.301 / 2 = −0.150. So α ≈ 0.15. That number is the whole story: a 100× increase in parameters bought you one halving of error. To halve the error again you'd need another 100×. That is the brutal arithmetic of scaling, summarized in a single exponent.

Let us make the exponent tangible. The widget below plots a power law two ways at once: on ordinary (linear) axes where it is a swooping curve that hugs a floor, and on log-log axes where it is a perfectly straight line. Drag the exponent α and watch the slope of the line change — steeper line means a bigger exponent means error drops faster per decade of resource. Notice that no matter the exponent, the linear-axis curve always flattens: that flattening is the diminishing returns we'll obsess over later, and it is exactly what a constant log-log slope looks like in linear space.

The same power law on linear axes (curve) and log-log axes (line)

Drag the exponent α. Left: error vs resource on ordinary axes — a curve that flattens. Right: the identical data on log-log axes — a straight line whose slope is −α. The two pictures are the same law; log-log just makes the exponent readable.

Scaling exponent α 0.16

One more idea that will save you from a classic mistake. A pure power law error = A·N−α predicts error eventually reaching zero as N grows without bound. That is wrong for real tasks — no model gets perfect, because data has noise and ambiguity. So practitioners use a shifted power law: error = E + A·N−α, where E (E-infinity) is the irreducible error — the floor you can never cross no matter how much you scale. Add that constant and the log-log line bends down toward the floor at large N instead of marching to zero. We have a whole chapter (6) on what sets that floor, but plant the idea now: the straight line is only straight above the floor.

Misconception: "a power law means I can keep halving error forever by scaling." No. First, the exponent is usually small (α in the 0.1–0.4 range for vision), so each halving of error costs a huge multiplicative jump in resource — and the cost compounds. Second, the irreducible floor E means the line is not really straight all the way down; it bottoms out. Reading a power law as "infinite free lunch" is the single most expensive misreading in the field. It is a precise diminishing-returns law, not a promise of perfection.
On a log-log plot of error vs parameters, you measure a straight line with slope −0.20. Parameters go from 1e6 to 1e9 (a 1000× increase, three decades). By roughly what factor does the error shrink?

Chapter 2: Scaling Vision Transformers — The ViT-G Curve

Now we ground the abstraction in the landmark vision result. In 2021, a team at Google Research published Scaling Vision Transformers — a systematic study of what happens to image-classification accuracy as you push Vision Transformers (ViTs — image models that cut a picture into patches and process them like words in a sentence) across two orders of magnitude in size and data. The headline artifact was ViT-G/14 ("G" for giant), a roughly two-billion-parameter model trained on about three billion images, which reached around 90.45% top-1 accuracy on ImageNet — a record at the time.

But the number is not the lesson. The shape is the lesson. When the authors plotted accuracy (really, error) against compute on log-log axes, they got the now-iconic saturating curve: error falls along a power law for a long while, then visibly bends and flattens as it approaches a floor. The straight-line power law of Chapter 1 is the middle of the story; the bend at the top is where the irreducible floor E starts to dominate. ViT-G is special not because it is huge but because it sits far enough out on the curve to show you the bend.

Two design findings from that paper are worth carrying. First, bigger models are more sample-efficient: a larger ViT reaches a given accuracy using fewer images than a smaller one, so if you have lots of compute, spending some of it on size pays for itself in data efficiency. Second, and crucially, data scale gates everything: the giant models only pull ahead when trained on enormous datasets. Train ViT-G on a small dataset and it underperforms a much smaller model — the data-starvation effect from Chapter 0, made concrete at billion-parameter scale. The authors needed a 3-billion-image internal dataset (JFT-3B) precisely because the model was big enough to exhaust anything smaller.

The key insight from the ViT-G curve: there are really two regimes on every scaling plot. In the power-law regime (the straight middle), error obeys E + A·C−α and every extra decade of compute buys a fixed fraction off the error — predictable, profitable. In the saturation regime (the flat top), error has approached E and extra compute buys almost nothing. Knowing which regime you are in is the most actionable thing a scaling plot tells you: scale aggressively in the first, and stop, distill, or change the problem in the second.

Let us put concrete numbers to the saturation. Model the ViT family as error = E + A·C−α with a floor E = 0.085 (an 8.5% error floor, i.e. a ~91.5% accuracy ceiling), A = 1.0, and α = 0.13. At a small compute C = 1 (in arbitrary units), error = 0.085 + 1.0·1−0.13 = 0.085 + 1.0 = 1.085 — well, capped at "very high." Step up: at C = 100, the power-law part is 100−0.13 = 10−0.26 ≈ 0.55, so error ≈ 0.085 + 0.55 = 0.635. At C = 10,000, it is 10,000−0.13 = 10−0.52 ≈ 0.30, error ≈ 0.385. At C = 1,000,000, it is 10−0.78 ≈ 0.166, error ≈ 0.251. Notice the gaps shrinking: each 100× in compute removes less error than the last, and all of them are creeping toward the 0.085 floor they can never cross.

Drag the compute slider below across the full ViT family — from a tiny Ti model up past ViT-G — and watch the point ride down the saturating curve. Early on, the marginal-gain readout shows fat returns; out near ViT-G it shows the returns nearly gone. This single picture is the empirical heart of "Scaling Vision Transformers."

Ride the ViT scaling curve from Ti to ViT-G

Slide compute across the Vision Transformer family. The curve obeys a shifted power law toward a ~91.5% accuracy ceiling. Watch the marginal gain collapse as you approach ViT-G — the bend is the whole point of the paper.

Compute (log scale, Ti → G) 0.30
Misconception: "ViT-G's 90.45% proves that scaling solves vision — just keep going." The saturating curve says the opposite. ViT-G sits in the bend, where the curve is nearly flat: the difference between ViT-G and a model with ten times more compute would be a fraction of a percent. The lesson of the curve is not "scale wins forever" but "scaling has a clear, measurable end-of-profit, and ViT-G is the model that showed us where it is." After the bend, the smart move is not more compute — it is better data, distillation, or a different task.
The ViT-G scaling curve, plotted on log-log axes, is a straight line in the middle but visibly bends and flattens at the high-compute end. What causes the bend?

Chapter 3: Compute-Optimal Allocation

Back to the whiteboard problem from Chapter 0, now with the tools to solve it. You have a fixed compute budget C. Training compute is, to a good approximation, proportional to model size times data size: C ≈ k · N · D, where N is parameters and D is the number of training examples (a common rule of thumb in the literature is C ≈ 6 · N · D for a single forward-backward pass over D tokens or images). The question: given C fixed, what split of N and D minimizes error? This is the compute-optimal allocation problem, and it is exactly what the Chinchilla paper famously solved for language models — with a lesson that transfers straight to vision.

Here is the intuition, then the arithmetic. Error falls as a power law in both N and D separately: roughly error = E + B·N−β + C'·D−γ, two power-law terms plus the floor. If you pour everything into N (huge model, tiny data), the D-term blows up — the model is data-starved. If you pour everything into D (tiny model, oceans of data), the N-term blows up — the model lacks capacity. The minimum-error split balances the two terms against each other, and because compute couples them through C = k·N·D, you can't grow one without shrinking the other at fixed budget.

The clean result — Chinchilla's headline — is that as you scale the compute budget, you should scale model size and data size together, roughly in proportion. Their finding for language models: for every doubling of compute, double the parameters and double the data. The earlier generation of models (like GPT-3) were, by this rule, badly undertrained — too big for the data they saw. The same principle governs vision: a compute-optimal ViT pairs each step up in size with a matching step up in images. Spending a windfall of compute entirely on parameters is the most common — and most expensive — allocation mistake.

Compute-optimal split, worked on a toy budget. Take error = N−0.5 + D−0.5 (drop the floor for clarity; equal exponents for symmetry), and a budget constraint N·D = 100 (so D = 100/N). Substitute: error(N) = N−0.5 + (100/N)−0.5 = N−0.5 + (N/100)0.5 = N−0.5 + 0.1·N0.5. Try a few splits: N=1 → 1 + 0.1 = 1.10. N=4 → 0.5 + 0.2 = 0.70. N=10 → 0.316 + 0.316 = 0.632. N=25 → 0.2 + 0.5 = 0.70. N=100 → 0.1 + 1.0 = 1.10. The minimum is at N=10, D=10 — the balanced split where the two terms are equal (0.316 each). With equal exponents, compute-optimal means equal resource. Push off-balance in either direction and error rises symmetrically. That valley is the whole idea.

The widget makes that valley visible. The curve is total test error as you slide the N/D split at a fixed budget; the marker sits where you put it, and the green line marks the compute-optimal minimum. Drag toward "all params" and watch error climb the data-starvation wall; drag toward "all data" and watch it climb the capacity wall. The bottom of the valley is where Chinchilla tells you to live.

The compute-optimal valley: where to split a fixed budget

Fixed budget, split between parameters and data. The U-shaped curve is total error; the green line is the compute-optimal minimum. Both walls (starvation, saturation) are steep — that's why getting the split right matters.

Split: all data ←→ all params 0.50
Misconception: "compute-optimal means the cheapest model that hits my accuracy target." It does not. Compute-optimal answers a training question: given a fixed training budget, what split minimizes error? It says nothing about inference cost. If you will serve the model to millions of users, you may deliberately over-train a smaller model — spend past the compute-optimal point on extra data for a model smaller than optimal — because a smaller model is cheaper to run forever. Chinchilla-optimal is the training sweet spot; deployment economics can rationally pull you off it.
Your compute budget for the next training run is going up 8×. Following compute-optimal (Chinchilla-style) scaling for vision, how should you spend it?

Chapter 4: Data Scaling — CLIP, LAION, and Why Quality Beats Count

So far "data" has been a single number D. Vision-language models broke that simplification wide open and taught the field its most important data lesson. CLIP (Contrastive Language-Image Pre-training) trained an image encoder and a text encoder jointly on 400 million image-caption pairs scraped from the web, learning to put an image near its caption in a shared space. The open reproduction, LAION, scaled this to two billion and then five billion pairs. And the punchline of years of follow-up work is blunt: past a point, raw count is the wrong axis. Data diversity and quality matter more than data quantity.

Why would more data ever not help? Web data is full of two poisons. The first is duplication: the same image (or near-identical crops) appears thousands of times. A billion-pair dataset might contain only a few hundred million distinct concepts; the rest is redundant copies that inflate the count without teaching anything new. The DataComp benchmark and the "no free lunch in data" line of work showed that aggressive deduplication — removing duplicates and near-duplicates — can improve a model trained on a smaller deduplicated set over a larger raw one. Counting pairs overstates how much you really have.

The second poison is noise and mismatch: web captions are often wrong, generic ("image"), or unrelated to the picture. A mislabeled pair is worse than no pair — it actively teaches a false association. CLIP-style training is somewhat robust to noise because of the contrastive objective, but only up to a point; the highest-performing models now run heavy quality filtering (CLIP-score thresholds, caption-image agreement, aesthetic and safety filters) before training. The frontier of data scaling is no longer "scrape more" — it is "curate better," and curation can beat collection.

The reframe that defines modern data scaling: the effective dataset size is not the number of pairs you downloaded — it is the amount of distinct, correctly-paired, diverse signal inside them. A deduplicated, quality-filtered set of 100M pairs can out-train a raw 400M set. So the data axis in your scaling law should be measured in effective examples, not collected ones. This is why DataComp made the dataset itself the object of study: at fixed compute, the data recipe became a bigger lever than the architecture.

Let us make the duplication math concrete, because it is startling. Suppose you scrape 400 million pairs but, due to web redundancy, only 25% are distinct — so 100 million unique pairs, each appearing four times on average. Your nominal dataset is 400M; your effective dataset is 100M. If error scales as Deff−0.3, then plugging 400M vs 100M into the law: the ratio of errors is (400M / 100M)−0.3 — but that 400M is a lie; the real error is set by 100M. You paid to download, store, and process 4× the data and got the model that 100M unique pairs would give you. Deduplication recovers that wasted 3× — same accuracy, a quarter of the compute, or far better accuracy at the same compute.

The widget below lets you scale a raw dataset while controlling its duplication rate and noise rate. Watch how the model's accuracy tracks the effective (distinct, clean) data, not the raw count. Crank duplication up and the accuracy curve flattens even as the raw count soars — you are spending on copies. Crank noise up and accuracy actively drops past a point — bad pairs teach bad associations. This is the empirical case for curation over collection, made tangible.

Raw count vs effective data: duplication and noise erode the scaling

Scale the raw dataset, then dial in duplication and noise. Accuracy follows the effective clean-and-distinct data (teal), not the raw count (dim). High duplication flattens the gain; high noise actively reverses it. Curation is a lever, not a chore.

Raw dataset size0.60
Duplication rate0.30
Noise rate (bad pairs)0.10
Misconception: "more data is always at least harmless — worst case it doesn't help." Wrong on two counts. Duplicated data costs compute and storage for zero accuracy gain (it is not free), and noisy data can make the model worse by teaching false image-text associations. The modern result is that a smaller, deduplicated, quality-filtered dataset routinely beats a larger raw one at the same compute. "Just scrape more" is the data-scaling version of the Chapter 0 mistake: confusing the count of a resource with its usable amount.
You scrape 1 billion image-text pairs, but 75% are duplicates or near-duplicates. A colleague proposes deduplicating down to the 250M distinct pairs before training. What is the likely effect on the trained model's accuracy at fixed compute?

Chapter 5: Emergent Zero-Shot & Transfer Scaling

Scaling laws would be a footnote if they only governed the pre-training loss. Their real power is that the benefits transfer: a model scaled up on a big generic dataset becomes better at tasks it was never explicitly trained on, and that downstream improvement also scales smoothly. Two transfer measurements dominate vision, and you must be able to read both: linear-probe accuracy and zero-shot accuracy.

Linear probe means: freeze the pre-trained model, run your labeled images through it to get feature vectors, and train only a single linear classifier on top. You are measuring "how good are the frozen features?" without changing the backbone. The beautiful empirical fact: linear-probe accuracy on a downstream task improves as a smooth power law in the pre-training scale. Bigger backbone, more pre-training data → better frozen features → higher linear-probe accuracy, predictably. You can fit that curve from a few model sizes and forecast the linear-probe accuracy of a backbone you haven't built.

Zero-shot is the CLIP magic trick: classify images with no task-specific training at all. Embed the image, embed the text "a photo of a {class}" for each candidate class, pick the nearest. Astonishingly, zero-shot accuracy also scales smoothly with pre-training scale — the larger and better-trained the CLIP model, the better it classifies categories it was never explicitly taught. This is the closest thing in vision to "emergence": a capability (recognizing arbitrary classes from a text prompt) that strengthens steadily as you scale, even though no one trained the model on those specific classes.

"Emergent" vs "smooth," worked carefully. In language models, some capabilities look like they appear suddenly past a scale threshold — flat, then a jump. Much of that apparent suddenness is now understood as an artifact of harsh metrics: if you score a task as all-or-nothing (exact match), a steadily-improving underlying ability crosses the "good enough to get it exactly right" line abruptly, looking like a jump. Measure the same ability with a smooth metric (per-token probability, or top-5) and the curve is a gentle power law all along. Vision transfer is mostly of the smooth kind: linear-probe and zero-shot accuracy rise predictably. The takeaway: when a capability looks "emergent," check whether it is the metric, not the model, that is discontinuous.

Let us quantify a transfer fit. Suppose across four CLIP scales you measure zero-shot ImageNet accuracy: at relative compute 1× you get 35%, at 10× you get 52%, at 100× you get 64%, at 1000× you get 72%. The error (1 − accuracy) is 0.65, 0.48, 0.36, 0.28. Take logs: each 10× of compute multiplies the error by roughly 0.48/0.65 ≈ 0.74, then 0.36/0.48 = 0.75, then 0.28/0.36 ≈ 0.78 — remarkably constant, the signature of a power law with slope ≈ log(0.75) ≈ −0.125 per decade. From those four cheap points you can extrapolate: at 10,000× compute, error ≈ 0.28 × 0.76 ≈ 0.21, so ~79% zero-shot — a forecast you can budget against before spending a cent on that run.

The widget lets you scale a CLIP-like model and watch both transfer curves rise together. The teal line is linear-probe accuracy; the warm line is zero-shot. Both climb as smooth power laws toward their ceilings, and both let you read off a forecast. Notice zero-shot starts lower but the gap narrows with scale — bigger models need less hand-holding to recognize new classes.

Transfer scales smoothly: linear-probe and zero-shot vs pre-training scale

Scale a CLIP-like model. Linear-probe (teal) and zero-shot (warm) downstream accuracy both rise as smooth power laws — no magic jump, just predictable transfer you can fit and forecast.

Pre-training scale (log) 0.35
Misconception: "zero-shot ability appears suddenly once a model is big enough — it's an on/off switch." For vision transfer this is mostly false. Zero-shot and linear-probe accuracy rise smoothly and predictably with scale; what looks like a sudden jump is usually a coarse pass/fail metric crossing a threshold over a steadily-improving ability. Treating transfer as a smooth power law — fittable from small runs — is both more accurate and more useful than waiting for a mythical emergence threshold.
You measure a CLIP model's zero-shot accuracy at three pre-training scales and see it improve along a smooth power-law curve. A teammate insists "zero-shot is emergent — it should jump suddenly, not curve smoothly." What is the best response?

Chapter 6: Diminishing Returns & the Irreducible Floor

We have hinted at it for five chapters; now we confront it head-on. Every scaling law has a floor — the irreducible error E — that no amount of parameters, data, or compute can cross. Understanding what sets that floor is what separates someone who scales blindly from someone who knows when to stop. The floor has several sources, and the dominant one in vision is humbling: label noise.

Label noise is error baked into the ground truth itself. ImageNet, the canonical benchmark, was labeled by humans, and humans disagree and make mistakes. Careful re-examinations found that several percent of ImageNet labels are wrong or genuinely ambiguous — a photo with both a dog and a person, labeled only "dog"; a category boundary (is it a "laptop" or a "notebook"?) that two reasonable annotators split differently. When the answer key is wrong a few percent of the time, a perfect model that gives the true answer is marked wrong on those examples. There is a ceiling on measured accuracy that has nothing to do with the model.

This is why ViT-G's ~90.5% is not embarrassing — it is bumping against a label-noise ceiling. Estimates put the "real" ceiling on ImageNet top-1 in the low-to-mid 90s precisely because of label issues; a model scoring 95% might be making more true errors than one scoring 91% if it has overfit to the mislabeled examples. The floor E in our shifted power law is, to a large degree, this irreducible label-noise term plus genuine task ambiguity (some images really are unidentifiable) and the Bayes error — the minimum error any classifier could achieve given the inherent overlap between classes.

The decomposition every scaling engineer should carry: measured error = reducible error (the A·N−α term you can buy down with scale) + irreducible error E (label noise + task ambiguity + Bayes error, which scale cannot touch). Money buys down the first term and nothing else. The single most important scaling decision is estimating E for your task, because it tells you the most accuracy money could ever buy. If you're at 88% and E implies a 91% ceiling, you have 3 points of headroom — not 12. Spending to chase the missing 12 is lighting money on fire.

Let us compute headroom by hand. Say your model is at 86% accuracy (0.14 error) and you estimate the irreducible floor at E = 0.08 (a 92% ceiling) from label-noise studies. Your reducible error is 0.14 − 0.08 = 0.06 — that 6 points is all scale can ever recover. Now suppose your scaling law says each 10× of compute removes 25% of the reducible error. After one 10×: reducible 0.06 → 0.045, accuracy 87.5%. After another 10×: 0.045 → 0.034, accuracy 88.6%. After a third (1000× total compute!): 0.034 → 0.025, accuracy 89.5%. Three orders of magnitude of compute — a thousandfold cost — bought 3.5 points, and you're still 2.5 points short of the ceiling, which you'll never reach. This arithmetic is the difference between a funded roadmap and a bankrupt one.

The widget below splits measured error into its reducible and irreducible parts as you scale. The green band is the irreducible floor you can never cross; the shrinking warm band is the reducible error you're buying down. Watch the warm band thin toward nothing while the green floor stands immovable — the picture of why "just scale more" eventually stops being a strategy.

Reducible vs irreducible: what scale can and cannot buy

Total error splits into a green irreducible floor (label noise + ambiguity, immovable) and a warm reducible part (buyable with scale). Slide compute up: the warm band thins toward the floor it can never cross. That gap is the end of scaling's profit.

Compute scale (log)0.30
Label-noise floor E∞0.08
Misconception: "if my model is below 95%, scaling can still get me there." Only if the irreducible floor allows it. On a benchmark with a 92% label-noise ceiling, no model — however large — will truly exceed ~92%; a reported 95% likely means the model has learned to reproduce the mislabels, which is worse, not better. Before committing to a scaling campaign, estimate E for your data. The reducible gap, not the gap to 100%, is your real budget. Cleaning labels can lower E itself — sometimes a far cheaper win than more compute.
Your model sits at 87% accuracy. Label-noise studies estimate the irreducible floor for this benchmark at 9% error (a 91% ceiling). Your scaling law removes 30% of the reducible error per 10× compute. Roughly how much accuracy can a further 100× compute realistically buy?

Chapter 7: ShowcaseFit a Scaling Law & Extrapolate

This is the payoff: a live scaling-law laboratory that does what a real research team does before a big run. You have a handful of noisy measurements from small, cheap models. You fit a power law to them in log space, draw the line with an uncertainty band, and extrapolate to a target scale you cannot afford to train — reading off both the forecast and how confident you should be in it. Every control teaches a chapter.

The dots are measured (error, scale) points, each carrying noise — just like real runs that vary with seed and data order. Slide measurement noise to make them scatter more; watch the fitted line stay put but the uncertainty band widen, because a noisy fit is a less certain forecast. Slide the number of data points: more pilot runs pin the line down tighter (narrower band), the empirical reason teams do several small runs before one big one. Slide the irreducible floor E to switch between a pure power law (straight forever) and a shifted one (bends toward the floor) — and watch how assuming the wrong floor wrecks the extrapolation.

Then drag the extrapolation target far to the right, past all your data. The vertical marker shows the forecast error at that scale, and the band around it shows your confidence. Three things to try, each a lesson made physical:

Extrapolate close to your data (target just past the last point): the band is tight — near-term forecasts are trustworthy.
Extrapolate far (target many decades out): the band fans out dramatically — a forecast 100× beyond your data is a guess with big error bars, which is exactly why you don't bet a whole budget on a single distant extrapolation.
Crank noise with few points: the band balloons — you've manufactured the situation where a scaling "law" is really wishful line-drawing through scatter, and the honest move is more pilot runs before committing.

Scaling-law lab: fit noisy points, see error bars, extrapolate to a target

Log-log axes. Warm dots = noisy pilot measurements. Teal line = the least-squares power-law fit. The shaded band = forecast uncertainty (widens with noise, narrows with more points, fans out as you extrapolate further). Drag the target past the data to read a budget-grade forecast with honest error bars.

Measurement noise0.05
Number of pilot points6
Irreducible floor E∞0.06
Extrapolation target (log scale)1.50
What you just proved to yourself: a scaling law is a straight-line fit in log space with an honest uncertainty band, and its forecasting power is real but bounded. Near your data the band is tight and you can budget confidently; far from your data, or with noisy/few points, the band fans out and the "law" is mostly hope. The discipline of scaling is not "draw a line and trust it to infinity" — it is "fit carefully, quote the error bars, and re-measure before betting big." That single panel is Chapters 1, 6, and the whole forecasting promise in one place.

If you removed this simulation, would you lose understanding? Yes — because "extrapolation uncertainty grows with distance" is an abstract sentence until you drag the target out and watch the band fan from a confident sliver into a fog. The napkin version of this entire lesson is exactly this picture: cheap points, a fitted line, a floor, and a forecast whose error bars tell you whether to trust it.

Chapter 8: Practical — Scale, Distill, Probe, or Fine-Tune?

You now understand the curves; here is how to act on them. In practice you rarely train a frontier model from scratch — you decide how to get accuracy given a model that already exists and a budget that is finite. Four moves cover almost every situation, and choosing among them is the real job. Scaling laws are the map; this is the compass.

Scale (train bigger / longer / on more data) is right when you are clearly in the power-law regime — far from the floor, with fat marginal returns — and you have both the compute and the effective data to feed a bigger model. The scaling-law check: fit your current runs, confirm you're on the straight part of the curve (not the bend), and verify a balanced N&D growth. If your marginal-gain readout is healthy, scaling is the highest-leverage move.

Distill (train a small "student" model to mimic a large "teacher") is right when you've already paid for a great big model but cannot afford to serve it. Distillation transfers much of the teacher's accuracy into a small, cheap-to-run student — you keep most of the quality at a fraction of the inference cost. This is the deployment-economics escape hatch from Chapter 3: compute-optimal training gives you a big model; distillation gives you something you can actually ship.

Linear-probe (freeze the backbone, train a tiny linear head) is right when you have a strong pre-trained model, limited labels (say under ~10K), and want strong results cheaply. From Chapter 5, frozen-feature quality scales with pre-training, so a linear probe on a big frozen backbone is often shockingly good for almost no compute — and it is robust to overfitting precisely because you're training so few parameters. Full fine-tune (update the whole backbone) is the last resort: right only when you have lots of in-domain labels and are chasing the final points, accepting the overfitting risk and cost.

The decision tree, as a single flow: Few labels? → zero-shot to scope, then linear-probe a frozen giant. Lots of labels, chasing last points?full fine-tune. Have a great big model but can't serve it?distill to a small student. On the straight part of the curve with compute and effective data to spare?scale — balanced N&D. Near the irreducible floor? → stop scaling; clean labels or change the task instead. The scaling law is what tells you which branch you're on; the tree tells you what to do about it.

Let us cost a real decision. You have a frozen ViT-G and 5,000 labeled images for a new task. Option A: full fine-tune ViT-G — gradients through 2B parameters, high overfitting risk on 5K images, large GPU bill, maybe 84% accuracy after careful regularization. Option B: linear-probe — one forward pass to extract 5,000 feature vectors, then fit a linear classifier in seconds on a laptop, maybe 82% accuracy at one-thousandth the cost and zero overfitting drama. Two points of accuracy for a 1000× cost difference and far less risk: on 5K labels, the linear probe is almost always the correct call. The scaling-law literacy is knowing that the frozen features are already excellent because the backbone was scaled, so you don't need to disturb them.

Pick the right move: accuracy vs cost across the four strategies

Slide your label budget. The bars show rough accuracy-per-cost for each strategy at that label count. With few labels, linear-probe dominates (cheap, no overfitting); only with many labels does full fine-tune pull ahead. Distill targets inference cost, not accuracy.

Labeled examples available 0.20
Misconception: "to get the best accuracy, always fine-tune the biggest model end-to-end." This is the most common and most expensive practical error in applied vision. With limited labels, full fine-tuning a giant overfits and underperforms a linear probe on its frozen features, at orders of magnitude more cost. The competent default in 2024+: zero-shot to scope, linear-probe a strong frozen backbone, distill if inference is the bottleneck, and only full-fine-tune when you genuinely have the labels and the headroom (check E!) to justify it.
You have a powerful frozen pre-trained backbone, only 4,000 labeled images for a new task, and a tight compute budget. Which move is usually best, and why?

Chapter 9: Connections & Where It Leads

Scaling laws are not a vision-only curiosity — they are a lens that, once you have it, reorganizes how you read the entire field. The exact pattern you built here — error falls as a power law in each resource, bends toward an irreducible floor, and the smart move is to estimate the floor and allocate the budget — recurs across language, audio, multimodal, and even reinforcement learning. Naming those connections turns one lesson into a map.

Up the size ladder. ViT-G was 2B parameters; the line continued to ViT-22B, a 22-billion-parameter Vision Transformer that pushed the curve further and stress-tested whether the engineering (training stability, sharding) keeps up with the scaling promise. The architecture that makes these models possible is the Vision Transformer, and the contrastive image-text training that powers CLIP-style data scaling is in Contrastive CLIP. The deep mechanics live in the veanors paper walkthroughs of ViT-G and ViT-22B.

Across to the metric side. The irreducible-floor and ambiguity ideas of Chapter 6 are the same forces that cap Scene Classification at ~61% — that lesson's Chapter 6 is the label-noise ceiling told from the task side; this lesson is it told from the scaling side. Read them together and you see one phenomenon from two angles.

Down to deployment. The Chapter-8 "distill instead of scale" move connects to the practice of making big models cheap to run; and the opposite extreme — what scaling laws look like when you have almost no compute — is the world of TinyML, where the entire game is squeezing accuracy out of kilobytes and milliwatts, the far-left end of every curve in this lesson.

Resource scaledPower-law termWhat gates itPractical move
Parameters (N)B·N−βNeeds matching data, or it starvesGrow with data (Chinchilla)
Data (D)C·D−γEffective data: dedup + quality, not raw countCurate over collect
Compute (C ≈ N·D)A·C−αDiminishing returns toward the bendCheck which regime you're in
(none) — the floorELabel noise + ambiguity + Bayes errorClean labels; estimate headroom
The one idea to carry out the door: scaling is an allocation problem with a known shape. Error falls as a power law — a straight line on log-log paper — in each of parameters, data, and compute; it bends toward an irreducible floor set by label noise and ambiguity; and the engineer's job is to fit that line from cheap pilots, estimate the floor, allocate the budget balanced (Chinchilla), measure data in effective not raw units, and stop scaling when the marginal gain dies. Master that and "how big should the model be?" stops being a guess and becomes arithmetic.

Cheat sheet

ThingWhat to remember
Power lawerror = E + A·N−α; straight line on log-log with slope −α
Fit itlog both sides, least-squares (polyfit) on (log N, log err); α = −slope, A = exp(intercept)
ViT-G~2B params, ~3B images, ~90.45% ImageNet; the model that showed the saturating bend
Compute-optimalC ≈ k·N·D; scale N and D together (Chinchilla); balanced split minimizes error
Data scalingEffective > raw: dedup + quality filter; noisy/dup data wastes or reverses gains (CLIP/LAION/DataComp)
TransferLinear-probe & zero-shot accuracy scale smoothly; "emergence" is often a harsh-metric artifact
Floor EIrreducible: label noise + ambiguity + Bayes error. Reducible gap, not gap-to-100%, is your budget
ActFew labels → linear-probe; can't serve big model → distill; far from bend → scale; near floor → clean labels

"It is the mark of an educated mind to rest satisfied with the degree of precision which the nature of the subject admits, and not to seek exactness where only an approximation is possible." — Aristotle, which is precisely the posture a scaling law demands: a power law forecasts a range, not a point, and the error bars are the honesty.