Scaling Laws for Native Multimodal Models

Bolting a vision encoder onto a language model is not better than training one transformer on images and text from scratch. Across 457 trained models, both reach the same loss.

Pick a design, then drag the compute budget. Watch the three loss curves run almost on top of each other, and the sparse one sit just below. Then we build, piece by piece, how the authors measured it: the loss law, the two architectures, the best split of a budget, the data, and the experts.

You need the idea of a loss (a number that says how wrong a model is, lower is better) and a feel for powers like 10 to the 22. We build the rest from zero.

Paper: Scaling Laws for Native Multimodal Models, Mustafa Shukor, Enrico Fini, Victor Guilherme Turrisi da Costa, Matthieu Cord, Joshua Susskind and Alaaeldin El-Nouby (Apple and Sorbonne University, 2025). An assigned reading of MIT 6.S985 Multimodal AI (Spring 2026, Paul Liang). It belongs to Lecture 8, Large Multimodal Models. Every idea is re-taught here in our own words and devices.

Spend a compute budget

The curves are the paper’s own fits of final validation loss L against training compute C in FLOPs (floating point operations, the count of arithmetic steps): early fusion L = 29.574 C−0.0492, late fusion L = 29.923 C−0.0494, sparse early fusion L = 27.086 C−0.048 (text of Section 4.2 and Figure 26). Loss is the average over three validation sets. The curves hold for the paper’s runs, which span about 1018 to 1022 FLOPs; the stretch beyond is extrapolation.

Chapter 0

Glue Or Grow? The Question Behind the Paper

See why a team trained 457 models just to ask whether the usual way of building a seeing, reading model is the best way

Imagine you want a program that looks at a photo of your fridge and answers, in words, “What can I cook tonight?” It must see, and it must write. Two kinds of skill, in one program.

Almost everyone builds it the same way. They take a ready-made vision encoder, a model that turns a picture into a list of numbers. They take a ready-made language model, a model that reads and writes text. They join the two with a small adapter, then train the pair a little more on pictures with captions. Think of it as gluing a pair of eyes onto a writer.

This glue works well. It also needs little new training, because both parts already know a lot. The paper asks a sharper question. Is the glue the best design, or only the convenient one?

The other way: grow one model from the start

The other way is to train one model from scratch, on pictures and words together, from its very first step. The paper calls this a native multimodal model (NMM). A modality is one kind of input, such as text or images. “Native” means the model never had a separate childhood as a pure reader or a pure viewer.

There is a reason to hope this is better. A model that was a pure language model first may carry habits that block it from using pictures fully. The authors name the worry: starting from single-mode training “may introduce biases that prevent the model from fully leveraging cross-modality co-dependencies”. A second worry is plain bookkeeping. Each glued part has its own training recipe and its own growth curve, so scaling up the whole thing means tuning several recipes at once.

There is also a reason for doubt. Training from scratch costs a lot. You cannot afford to try both designs at every size by guesswork. You need a way to predict how each design behaves before you pay for the big run. That way is called a scaling law.

What a scaling law is, in plain words

Train a small model and record its validation loss: a score of how badly it predicts the next piece of held-out text or image-text, where lower is better. Train a model twice as big and record again. Do it at many sizes and many amounts of training data. When you plot the scores, they fall along a smooth, bendy line. A scaling law is a short formula for that line.

Why care about a formula? Because it lets you draw the line for a design you can only afford to test small, then read off what a huge run will give. It also tells you how to spend a fixed budget: bigger model, or longer training? Chapters 1 to 3 build this formula from nothing.

What the authors actually did

They trained 457 models. The sizes run from 0.3 billion to about 4 billion active parameters (the adjustable numbers inside a model, which training sets). The training length runs from 250 million to 400 billion tokens for the dense models, and up to 600 billion for the sparse ones (the pieces a model reads: a word piece, or a patch of a picture). They varied the design, the mix of data, and later the use of experts. Everything was measured the same way, so the curves can be compared.

Their answers, in the order we meet them:

  1. Early fusion matches late fusion. With the same compute, training from scratch with no vision encoder reaches the same loss, and is a little better when compute is small.
  2. The two designs spend a budget differently. Late fusion wants more parameters. Early fusion wants more training tokens.
  3. Multimodal models scale like language models. The exponents sit in the same range as those for text-only models.
  4. Experts help. A sparse early-fusion model beats a dense one at the same running cost, and its experts sort themselves by modality without being told to.

A picture becomes tokens: the cost you will see all lesson

One fact about pictures matters from the start. A transformer reads a list of tokens, not a grid of pixels. So the model cuts the image into small squares called patches. Each patch becomes one token. The paper uses 224 by 224 pixel images cut into 14 by 14 patches, so each side holds 224 / 14 = 16 patches, and the image becomes 16 × 16 = 256 tokens.

The models read at most about 1k tokens at once (the paper’s context length, which we count as 1,024). So one picture takes a big bite of the page. Drag the image size and watch the bite grow.

Cut the picture into tokens

Left: the image, cut into 14 by 14 pixel patches. Right: the 1,024 slots of the context, with the picture’s tokens in pink and the room left for text in grey. Drag the image side.

224

The patch size (14 by 14), the 224 pixel image and the 1k context come from the paper’s setup (Section 2.3); we count 1k as 1,024. Other image sizes are shown only to make the cost visible; the paper trains at 224. The paper lists high resolution as an open problem for early fusion (Appendix D) because the token count grows with the square of the side.

At 224 pixels the picture takes 256 of 1,024 slots: one quarter. At 448 pixels it takes all 1,024, and no text fits. The cost grows with the square of the side. Hold on to that: in the late-fusion design, the vision encoder pays this cost in its own separate network.

The road

This lesson follows the paper’s own order, with the background it assumes taught inline, and each chapter ends with a quiz on its one idea.

  1. The loss law: loss as a floor plus two shrinking gaps, and the compute rule C = 6ND.
  2. Two ways to join the senses: early and late fusion, with every tensor shape.
  3. Spending a budget well: fitting the law and deriving the best model size.
  4. Early against late: the loss curves, the parameter appetite and the training cost.
  5. What you feed the model: three data types, four mixtures, and scratch against warm start.
  6. Experts in the feed-forward layer: how a mixture of experts routes tokens.
  7. Sparse results and expert specialization: what experts buy and what they learn.
  8. The paper on one page: numbers, code, limits and where to go next.
What does the paper ask about the usual way of building a model that sees and reads?

Chapter 1

The Loss Law: A Floor and Two Shrinking Gaps

Write the final loss of a model as one short formula in its size and its training data, and learn what each part of it means

You are learning to throw darts. After a week you miss the bullseye by 30 cm on average. Practice another week and you miss by 20. Another, and 15. The misses keep shrinking, but each extra week buys less. And a floor exists: your hand shakes a little, so you will never hit 0 cm.

A model’s loss behaves like that. This chapter writes the shape in one line.

Two ways to get better, and a floor

A model can improve in two ways. It can get bigger: more parameters, written N. Or it can read more: more training tokens, written D. Both cost computing work, counted in FLOPs, the number of simple arithmetic steps (adds and multiplies).

There is a rule of thumb for the work. For each token, the model does about 2 steps per parameter going forward (one multiply, one add) and about 4 going backward to learn. That is 6 in all, so the total work is

C
The compute budget: total FLOPs spent training the model.
N
The number of parameters. For a mixture of experts, only the active ones, the parameters a token actually uses (Chapter 6).
D
The number of training tokens the model reads.

The paper uses this rule for early fusion. For late fusion it adds the vision encoder’s own work: C = 6(NvDv + ND), where Nv and Dv are the encoder’s parameters and the image tokens it reads (Table 1). Work out an example: 1 billion parameters reading 300 billion tokens costs 6 × 109 × 3 × 1011 = 1.8 × 1021 FLOPs.

The formula

Following earlier language-model studies (Hoffmann et al., Kaplan et al.), the paper writes the final validation loss as a floor plus two gaps:

L
The final validation loss: the average over three held-out sets, image-caption, interleaved and text-only (Table 1).
E
The floor: the lowest loss reachable on this data, even for an endless model trained on endless data. Language is partly random, so some loss cannot go away.
A / Nα
The size gap: what you lose because the model is too small. It shrinks as N grows. The constant A sets its starting height; α sets how fast it falls.
B / Dβ
The data gap: what you lose because the model has read too little. It shrinks as D grows, at the rate β.

A shape like 1 / Nα is a power law: the quantity changes by a fixed factor every time N changes by a fixed factor. For early fusion the paper measures α = 0.301 and β = 0.335 (Table 2, row AVG), with a floor E = 1.904. Check what α means. Multiply N by 10 and the size gap is multiplied by 10−0.301 = 0.50. Ten times the parameters halves the size gap. Ten times the tokens multiplies the data gap by 10−0.335 = 0.46.

Play with the formula

The paper’s tables give E, α and β. The constants A and B are not in the paper’s text, so the device below uses toy values, A = 200 and B = 2,000, picked so that the loss lands in the range the paper plots (about 2.4 to 3.0). Everything else is the paper’s. Drag the size and the data and watch the three parts of the loss.

Build a loss from three parts

The bar is the loss. Grey is the floor, yellow is the size gap, pink is the data gap. Drag the parameters and the tokens, or press a doubling button and watch which gap gives back more.

E = 1.904, α = 0.301 and β = 0.335 are the paper’s early-fusion fit (Table 2, AVG). A = 200 and B = 2,000 are illustrative, chosen so the loss falls in the paper’s plotted range. Compute uses C = 6ND.

Take the starting point, 1 billion parameters and 300 billion tokens. The size gap is 200 / (109)0.301 = 0.39. The data gap is 2,000 / (3 × 1011)0.335 = 0.29. The loss is 1.904 + 0.39 + 0.29 = 2.58. Now double N. The size gap falls by the factor 2−0.301 = 0.81, from 0.39 to 0.32, and the loss to 2.51. Double D instead, and the data gap falls by 2−0.335 = 0.79, from 0.29 to 0.23, and the loss to 2.52. Each doubling doubles C. Neither is free, and neither is the clear winner. Chapter 3 asks how to choose.

Where the formula can fail

Symptom. A formula fitted on small runs can be right where it was fitted and wrong far outside. Measurement. The authors compare the loss the formula predicts with the loss each run really reached, and they hold out a bigger model: they fit on small models and predict an 8-billion-parameter one (Appendix C.6, Figures 23 and 24). They report that the fit is accurate, especially at high compute, and that an 8B model is predicted with reasonable accuracy. Fix. Read the formula as a map of the region measured, not a law of nature. The paper itself lists extrapolation to extreme sizes as open (Section 7).

In L = E + A/Nα + B/Dβ, why can you not drive the loss to zero by making N enormous?

Chapter 2

Two Ways to Join the Senses

Follow one image and one sentence through an early-fusion model and a late-fusion model, and see every tensor shape on the way

Two kitchens make the same dish. In the first, a specialist looks at the fridge and writes a note: “eggs, spinach, feta.” She hands the note to a cook, who never sees the fridge. In the second kitchen, one cook looks at the fridge and cooks.

The first kitchen is late fusion: the pictures are handled by their own network, and the two streams meet deep in the pipeline. The second is early fusion: one network sees everything from its first layer. The paper compares the two with great care. First we need both, in full detail.

The shared body: a decoder transformer

Both designs end in the same engine, a decoder. A decoder is a transformer that reads a list of token vectors and, at each position, predicts the next token. Inside, each layer mixes tokens with attention (every token looks at others and takes a weighted blend of what it sees) and then reshapes each token on its own with a feed-forward block. The paper’s decoders use a feed-forward block called SwiGLU and a stabilizer on attention called QK-Norm, and every model has 24 layers. Only the width grows with size: the number of values in each token vector, written d.

Width d is a tensor dimension you will see in every shape below. Table 6 gives it for each model: 800 for the 275M-parameter model, 1,088 for 468M, 1,632 for 932M, 2,208 for 1.63B, 2,624 for 2.28B and 3,232 for 3.35B.

Early fusion: one linear layer is the whole eye

Here is the early-fusion data flow, with shapes. The image is 224 × 224 pixels with 3 colour channels: shape (224, 224, 3). It is cut into 14 × 14 patches, giving 16 × 16 = 256 patches. Each patch holds 14 × 14 × 3 = 588 numbers. Flattened, the image is (256, 588).

Now comes the only image-specific part. A single linear layer, one matrix of shape (588, d), multiplies every patch. That maps each patch to a vector of width d: the image is now (256, d), the same kind of object as a row of word vectors. For the 1.63B model, d = 2,208, so this matrix holds 588 × 2,208 ≈ 1.3 million numbers, about 0.08% of the model. The paper calls this using “almost no modality-specific parameters”.

Text takes the usual road: token ids (T) become vectors (T, d) through an embedding table. The two are laid end to end into one sequence of shape (256 + T, d) and fed into the decoder. No image tokenizer turns pictures into discrete codes. The image patches stay continuous numbers.

Late fusion: a separate eye that is itself a transformer

In late fusion the patches first go through a whole vision encoder: another transformer, built like CLIP (a well-known image-and-text model), with its own weights Nv and its own depth. It turns (256, 588) into a stack of 256 vectors of its own width, which are connected into the decoder’s input at width d. After that the decoder runs as before.

So a late-fusion model has more depth in total: encoder layers plus decoder layers. The paper notes that, for the same FLOPs, late-fusion models have a higher parameter count and a higher effective depth. It also makes the FLOP count differ, which is why C = 6(NvDv + ND) for late fusion, against 6ND for early.

The attention mask: who may look at whom

One more choice is hidden in early fusion. Text is written left to right, so a text token may look only at tokens before it: that is causal attention. An image has no left to right, so its patches may look at each other freely: bidirectional attention. The paper uses causal attention for text and bidirectional attention for image tokens, and reports that this works better. A mask is the grid of allowed looks, with one row per token that looks and one column per token looked at.

Trace the data flow

Pick a view. Each box is a step; the yellow shape on its right is what leaves it. Slide the model size to change the width d. The third view draws the attention mask.

Widths, the 24-layer depth, the 14 by 14 patch and the 224 pixel image are from the paper (Table 6, Section 2.3). T is the number of text tokens in the sequence; the paper’s sequences total about 1k tokens. The late-fusion connector’s exact form is not given in the text we use, so the box says only that it maps to width d. The mask grid is a three plus three toy.

Read the early-fusion view, then the late-fusion view. The early path has one stage between the image and the decoder, a matrix. The late path has a full network there. That difference is the heart of the experiment. Everything else, the data, the loss, the optimizer, was matched.

Training and inference

Training. A training sample is one of three kinds of sequence (Chapter 5): an image with a caption, a web page with images between the text, or plain text. The model runs one forward pass over the whole sequence. The loss is the standard next-token cross-entropy on the text tokens: how much probability the model gave to the true next word. The paper says its models “rely exclusively on a next-token prediction loss”. Images are inputs, never targets.

Inference. The image goes in once, as 256 vectors. The model then writes text one token at a time, as any language model does. Early fusion has one network to serve. Late fusion has two, and runs the encoder first.

Where the comparison can go wrong

Symptom. A late-fusion model trained naively looks worse than it should, so early fusion would seem to win for the wrong reason. Measurement. The authors tune the late-fusion recipe. They found that a smaller learning rate for the vision encoder significantly boosts performance (Table 8), and that freezing the encoder works best only when the encoder starts from a pretrained model (Table 7). Fix. They give late fusion its best recipe in every comparison, so any gap in the results is not a tuning gap.

In the paper’s early-fusion model, what sits between the raw image patches and the decoder?

Chapter 3

Spending a Budget Well: The Compute-Optimal Split

Derive how to split a fixed compute budget between model size and training tokens, then see how the authors fit the law that makes it possible

You have a fixed sum to spend on a road trip: say 1,000 euros. Rent a fancy car and you cannot afford many days. Take a cheap car and you can drive for weeks, but the trip is dull. Somewhere between the two is the split that gives you the best trip for the money.

Training a model is the same. Compute C is the money. A bigger model (N) is the fancy car. More tokens (D) are more days on the road. Since C ≈ 6ND, a bigger N forces a smaller D. The question is the best split. A model trained with that split is compute-optimal.

The valley

Fix the budget C. Now each choice of N fixes D = C / (6N). Put that in the loss law from Chapter 1 and the loss depends on N alone:

L(N)
The loss you get at this fixed budget when you choose a model of N parameters.
A N−α
The size gap. It falls as N grows.
B (6N / C)β
The data gap, with D replaced by C / 6N. It rises as N grows, because a bigger model leaves fewer tokens to read.

One term falls and one rises, so the loss is a valley. Too small a model and the size gap is large. Too large a model and the data gap is large. The bottom of the valley is the best choice, N*. The device below draws this valley. Move the budget and the valley slides. Then move your own model size away from the bottom and see what it costs.

Find the bottom of the valley

The curve is the loss you get at the chosen budget as you change the model size (left is smaller). The green dot is the best size. The teal dot is your choice. Slide the budget, then your size.

The exponents are the paper’s: multimodal early fusion from Table 2 (E = 1.904) and the text-only Chinchilla fit it quotes (E = 1.693). A = 200 and B = 2,000 are illustrative for both, so the sizes shown are illustrative; the shape of the valley, and the fact that the best size scales as Cβ/(α+β), do not depend on A and B.

The closed form

The bottom of a valley is where the slope is zero. Take the derivative of L(N) and set it to zero: −αAN−α−1 + βB6βNβ−1C−β = 0. Move the terms to opposite sides and multiply by N:

N*
The compute-optimal number of parameters for budget C.
αA Cβ / (βB 6β)
A constant times C to the power β. All the dependence on the budget sits in Cβ.

Take both sides to the power 1 / (α + β). Then N* grows as C to the power β / (α + β). Since D* = C / 6N*, the best token count grows as C to the power α / (α + β). The two exponents add to 1. They are the paper’s a and b.

Put in the early-fusion numbers. α + β = 0.301 + 0.335 = 0.636. So a = 0.335 / 0.636 = 0.527 and b = 0.301 / 0.636 = 0.473. The paper’s appendix gives 0.52649 and 0.47351 for the closed form, and the main text gives 0.526 and 0.473 from a direct fit. Double the budget and the best model grows by 20.527 = 1.44 times, while the best token count grows by 20.473 = 1.39 times. Params and tokens should grow almost in step. For a text-only model (Chinchilla’s α = 0.339, β = 0.285) the same formula gives a = 0.285 / 0.624 = 0.457, close to the 0.46 the paper quotes.

The paper draws the lesson in its own words: parameters and tokens should be scaled about equally. And since a = 0.526 is larger than 0.46 for text, the best multimodal model is somewhat bigger, and trained on somewhat fewer tokens, than a text model at the same budget.

How the paper fits the law

The closed form needs α, β and the constants. These must be fitted from the 457 runs. The recipe has three parts.

  1. Write the loss in logs. The law is a sum of three terms. Its log is a log-sum-exp: log L = LSE(a − α log N, b − β log D, e), where A = ea, B = eb and E = ee. This keeps A, B and E positive and keeps the numbers well scaled.
  2. Minimize a robust error. The objective is the sum over runs of Huber(LSE(…) − log Li), with δ = 10−3. The Huber loss is squared for small misses and only linear for large ones, so one odd run cannot drag the whole fit. The minimizer is L-BFGS, a standard routine for smooth problems.
  3. Start from many places. The surface has many valleys, so they try a grid of starting points (α, β in {0, 0.5, 2.5}; a, b from 0 to 30; e in {−1, −0.5, 0.5, 1}) and keep the fit with the lowest error.

They also check the answer a second way. For each budget C they compute the loss at log-spaced token counts, find the best N by reading off the minimum, and fit a power law through the (C, N*) pairs. The two routes agree closely (Appendix C.2, Table 9).

How sure are the numbers?

Symptom. A fitted exponent can wobble if you refit on slightly different runs. Measurement. The authors resample their runs with replacement 100 times and refit (Table 11). The spread of α is 0.101 and of β is only 0.029. Fix. There is no cure but honesty: β is better pinned down because the study used more token counts than model sizes, so trust the token side more. The paper adds that the fit improves with higher FLOPs.

If α and β were equal, how should you split a budget that you double?

Chapter 4

Early Against Late: What the Curves Show

Read Figures 1 to 4 and Table 2 line by line, and see that the two designs reach the same loss by spending a budget differently

Two runners finish a race in the same time. One took long strides. The other took short, quick ones. The finish line cannot tell them apart, but their shoes wear out differently. The same thing happens here. The two designs end at nearly the same loss, but they spend parameters and tokens in different amounts, and the difference changes what they cost to train and to serve.

Same loss, almost the same exponent

For each design the authors collect all their runs and take the lowest-loss frontier: for each compute budget, the best loss any run reached. A straight line on a log-log plot fits this frontier. The slope is the exponent c in L ∝ Cc.

Early fusion gives c = −0.0492. Late fusion gives c = −0.0494. The two numbers differ in the fourth decimal place. The text-only reference they cite, GPT-3, has −0.048. So adding images, and gluing or not gluing, does not change how quickly loss falls with compute. The hero at the top of this page draws these lines.

The constants in front differ even less: 29.574 for early and 29.923 for late. At 1022 FLOPs both predict a loss of about 2.45. This is the paper’s first headline: “no inherent advantage to late-fusion architectures”.

Where early fusion pulls ahead

Look closer, and the curves are not identical. Figure 3 trains models of many sizes on many token counts and plots loss against FLOPs. Early fusion has a small edge, mostly at small model sizes. The gap shrinks as the model grows, and the appendix says the gap closes mainly because of model size, not because of more tokens (Appendix B.1). Across all models the authors trained, early fusion was never beaten.

Three more checks point the same way. Raise the image resolution, and early fusion gains more (Appendix B.3). Raise the share of text-only data, and the gap in favour of early fusion widens (Figure 16). Change the mixture of data, and early fusion does best when interleaved and text-only data grow (Figure 7).

Different appetite for parameters

The sharper difference is in the best split from Chapter 3. For late fusion the best model is larger; for early fusion the best token count is larger. The paper shows it as the ratio N/D, parameters per training token, at the compute-optimal point:

Both ratios rise with the budget, but late fusion rises faster. At large budgets a late-fusion recipe asks for more parameters per token, and so a bigger network to hold and to run. The paper says this is “crucial for reducing inference costs and, consequently, lowering serving costs after deployment”. The cost of serving a model depends on the parameters it uses for each token, so fewer is cheaper every day the model runs.

The device below turns the two exponents into a picture. Only the slopes are the paper’s; the curves are pinned to the same starting height at 1018 FLOPs, an illustrative choice, so you compare how fast each ratio grows.

Compare the parameter appetite

Each line is parameters per training token at the best split, relative to its own value at 1018 FLOPs (log scale, so equal steps up are equal multiples). Drag the budget and read the ratios.

Exponents from the paper: early 0.053 and late 0.076 (Section 3.1), sparse −0.312 (Figure 1, bottom). Anchoring all three at 1 when C = 1018 is illustrative: the paper does not give the absolute ratios in the text. The sparse model counts active parameters only.

At 1022 FLOPs the gap between late and early is 10(0.076 − 0.053) × 4 = 100.092 = about 1.24. Late fusion wants about 24% more parameters per token than early fusion at that budget, by the pinned comparison. At 1024 the factor reaches about 1.37.

It also trains faster

Figure 4 measures it on 16 H100 GPUs for 160,000 steps (300 billion tokens). Early-fusion models use less memory per GPU and take fewer hours, under the same FLOP budget. The advantage grows as compute grows. The reason is bookkeeping: for the same FLOPs, late fusion carries extra parameters and extra effective depth, because the encoder layers sit alongside the decoder layers. The paper gives relative bars rather than a table of hours, so we do not quote a number.

The headline numbers on one table

MeasureEarly fusionLate fusionSparse early
Loss against compute29.574 C−0.049229.923 C−0.049427.086 C−0.048
α (size)0.3010.29030.710
β (tokens)0.3350.33830.372
Best size grows as Caa = 0.526a = 0.636a = 0.361
Best tokens grow as Cbb = 0.473b = 0.462b = 0.656
Parameters per token, N/DC0.053C0.076C−0.312

Table 2 (rows AVG) and Figure 1 of the paper. The sparse column is Chapter 7.

When the comparison could mislead

Symptom. The late-fusion line in Figure 3 sits above the early-fusion line for small models, but a reader may suspect a weak late-fusion recipe. Measurement. The paper varies the late-fusion setup itself: a smaller or larger encoder (Appendix B), a fixed encoder size (Figure 19), a pretrained encoder (Figure 21), different image resolutions. Fix. The same story holds in each: no setting gives late fusion an advantage that early fusion lacks. What the authors cannot test is every possible late-fusion design, so “no inherent advantage” is a claim about the designs they ran.

Early and late fusion reach almost the same loss at a given compute budget. What do the paper’s compute-optimal models differ in?

Chapter 5

What You Feed the Model: Three Data Types and Four Mixtures

Learn the three kinds of training data, see how the mixture tilts the compute-optimal split, and compare training from scratch with a warm start

A cook who learns from recipe cards alone cannot read a cookbook chapter. A cook who learns only from chapters never learns what the dish looks like. Good training needs a mix of sources. A model that sees and reads is no different, and the mix matters enough to change the scaling law.

The three data types

The paper trains on three kinds of sequence. Each teaches something different.

Table 5 lists the sources and how often each is sampled. Image captions come from DFN (2 billion samples, drawn 27% of the time), COYO (600 million, 11.25%) and a private set of high-quality image-text pairs called HQITP (400 million, 6.75%): 45% in all. Interleaved documents come from Obelics (141 million documents, 45%). Text-only comes from DCLM (6.6 trillion tokens, 10%). So the default mixture, written 45-45-10, is 45% captions, 45% interleaved and 10% text.

For each type the authors keep a held-out validation set, and the loss L in all earlier chapters is the average over the three. They also fit a separate law per data type. The result (Table 2) shows that the model learns captions fastest: the loss exponent is −0.061 for captions, −0.046 for interleaved documents and −0.042 for text. The paper does not say why; a likely reason is that a caption is short and its picture is easy to name.

Four mixtures, 76 models each

Do the findings depend on the mix? The authors test four, drawn from what other projects use, written caption-interleaved-text: 45-45-10 (the default), 40-20-40, 30-30-40 and 20-40-40. For each they train 76 models and refit everything.

The big picture: all four follow similar scaling lines, with the loss exponent between −0.0463 and −0.0492. But the best split moves. Table 3 lists the exponents. More caption data means a lower a (model size grows more slowly) and a higher b (tokens grow more quickly). More text and interleaved data does the opposite.

Why? The paper’s reading: caption data holds more image tokens than text tokens. Look at the I/T column, the ratio of image tokens to text tokens: 1.19 for 45-45-10, but only 0.49 for 20-40-40. When image tokens are plentiful, “training for longer decreases the loss faster than increasing the model size”. That is the authors’ reading of the pattern, not a separate test.

Tilt the split with the mixture

The top bar is the mixture. Below it, the two bars are how fast the best model size and the best token count grow with compute. The dashed line is an even split at 0.5. Pick a mixture.

Shares and exponents are from Table 3 (early-fusion models, mixtures written caption-interleaved-text). Doubling factors are 2a and 2b, computed here. The paper warns that broader mixture sweeps are needed before treating the coefficients as fixed (Section 7).

Take the default, 45-45-10. Here a = 0.527 and b = 0.474. Double the budget, and the best model grows 20.527 = 1.44 times while the best token count grows 20.474 = 1.39 times. Now take 20-40-40. Here a = 0.582 and b = 0.423, so the model grows 1.50 times and the tokens only 1.34 times. A text-heavy mix wants more of the budget in parameters.

From scratch or from a warm start?

A last data question links back to Chapter 0. Is it wasteful to train natively when strong language models already exist? The authors test it. They initialize an early-fusion model from DCLM-1B, a text model trained on over 2 trillion tokens, and continue on multimodal data. They compare it with a model trained from scratch (Figure 8).

The native model closes the gap. On image-caption data it matches the warm start after fewer than 100 billion multimodal tokens. On interleaved and text data it may need up to 1 trillion tokens. The authors weigh it against the cost of the warm start’s own pretraining, and conclude that native training can be a more efficient way to reach the same performance. Read that carefully: it is a statement about total cost, not about the warm start being bad.

The mixture hides a problem

Symptom. The average loss improves at one rate, but each data type improves at its own rate, so a single average law can hide where the gains come from. Measurement. The authors fit a law for each type (Table 2, Appendix C.4). They find that raising the share of a type raises its own exponent: moving captions from 30% to 40% lifts its exponent from 0.056 to 0.061 (Appendix C.5). Text-only is the exception, and its exponent barely moves. Fix. Report the per-type laws next to the average, and treat the average as a summary, not the whole story.

In the paper’s mixture study, what happens to the best token count as the share of image-caption data grows?

Chapter 6

Experts in the Feed-Forward Layer

Learn how a mixture of experts sends each token to one specialist, why it keeps the cost per token flat, and what stops the specialists from sitting idle

A hospital has one general ward where every patient is treated by the same doctors. It also has a version with eight specialist wards, and a triage nurse who looks at each patient and sends them to one ward. Each patient still sees only one ward, so each patient costs the same time. But the hospital as a whole knows eight times as much.

That is a mixture of experts (MoE). The paper uses it because pictures and words are different kinds of material. It would be natural for a model to keep some of its weights for each. Late fusion hard-wires that split. Early fusion with experts lets the model learn a split.

The part of the layer that gets replaced

Recall the decoder layer from Chapter 2: attention, then a feed-forward block (FFN). The FFN treats each token separately: it expands the token vector, passes it through a nonlinear squashing step, and shrinks it back. Most of a transformer’s parameters sit in these blocks. An MoE changes only this part. Attention stays as it was.

In a sparse layer the single FFN is replaced by E copies, the experts, each with its own weights. The paper uses E = 8. A small router decides which expert gets each token. The router is one linear layer of shape (d, 8). It turns each token vector into 8 scores, and a softmax (a squashing step that makes the scores positive and sum to 1) turns the scores into probabilities. Each token picks its top k experts. The paper uses k = 1, so each token goes to exactly one expert.

Follow the shapes through one layer. Tokens arrive as (T, d). The router gives scores (T, 8). The best-scoring expert for each token gives an index list (T). The tokens are gathered by expert: expert e receives a block (ne, d), where the ne add up to T. Each expert applies its FFN to its block. The results are scattered back to their original order, shape (T, d) again.

Cheap per token, large in total

Because a token uses one expert, the work per token is the work of one FFN, the same as a dense layer. The weights stored are eight FFNs. So an MoE separates two numbers a dense model keeps tied together: the active parameters, the ones a token touches, and the total parameters, everything stored.

A worked example with typical sizes (not the paper’s). Let the width be d. Attention holds about 4d2 weights and a SwiGLU FFN about 8d2, so a dense layer holds about 12d2. An 8-expert layer stores 4d2 + 8 × 8d2 = 68d2 but each token touches 4d2 + 8d2 = 12d2. The stored size is about 5.7 times the active size, and the compute per token is unchanged.

This is why the paper counts N as active parameters when it applies C = 6ND, and why it says sparse and dense models can be compared “at the same inference cost”.

Keeping every expert busy

A router that learns on its own can fall into a rut. A lucky expert gets a few more tokens early, so it trains a little more, so it gets better, so the router sends it still more. The rest starve. This is expert collapse. The paper adds an auxiliary load-balancing loss with weight 0.01, an extra term in the loss that grows when tokens spread unevenly. (One common form, from earlier work, is E ∑ fiPi, where fi is the share of tokens that go to expert i and Pi is its average router probability. It equals 1 for a perfectly even load, and 8 if one expert takes everything. The paper cites the idea but does not print its formula.)

A second hazard is token dropping. In many MoE systems each expert has a fixed capacity, and tokens that arrive after it is full are thrown away. The paper uses a dropless design (from Gale et al.) that never drops a token during training. For runs shorter than 20,000 steps it also uses a longer warmup, 2,500 steps, which it found “significantly better” for MoEs.

Who sends the tokens: rules or a router?

There are two ways to decide. Modality-agnostic routing gives the job to a learned router that does not look at the modality label. It sees only the token vector. Modality-aware routing uses a fixed rule: image tokens go to an image FFN and text tokens go to a text FFN. The paper compares them (Chapter 7). Try both below, on a toy batch of 24 tokens.

Route the tokens

Top row: 12 image tokens (pink) and 12 text tokens (grey). Bottom: the eight experts, each stacking the tokens it receives. Choose how tokens are sent, then press Route a new batch.

A toy: the router scores are seeded random numbers with a mild lean (image tokens toward experts 1 to 4, text toward 5 to 8), not a trained router. The setups follow the paper’s definitions of modality-agnostic and modality-aware routing (Section 2.1) and its balance-loss idea (Section 4); the numbers are illustrative. In the fixed setup only two FFNs exist.

In the learned setup the load is close to even and most experts see both kinds of token. In the fixed setup, only two FFNs work, and each sees one modality. In the collapsed setup one expert takes most tokens and the rest sit idle. That last picture is what the 0.01 loss term prevents.

The setup in one place

The paper’s sparse models are built on the dropless-MoE code of Gale et al. They use E = 8 experts and top-k routing with k = 1, a load-balance weight of 0.01, active parameters from 0.3B to 3.4B, and the early-fusion body from Chapter 2. The paper says it found this configuration to “work effectively”; the main text reports no sweep over E or k. Appendix E.1 (Table 13) lists a small sample of finetuning results: average score 40.1 for 4 experts with top-1, 41.7 for 8 experts with top-1 and 42.9 for 8 experts with top-2. So the choice is a tuning knob, and a wider top-k scored higher in that sample.

Why does an 8-expert, top-1 layer cost about the same FLOPs per token as a dense feed-forward layer?

Chapter 7

Sparse Results and Expert Specialization

Read Figures 9 to 13 and Table 4 to see what experts buy, how they change the best split, and what they learn to do on their own

Chapter 6 built the machine. Now the three questions a skeptic asks. Does it beat a dense model at the same running cost? How does it change the scaling law? And did the experts really learn to specialize, or is the story a guess?

Same cost, lower loss

Figure 10 trains dense and sparse early-fusion models at matched active parameters (275M, 464M, 932M and 1.63B) and many token counts. The sparse models beat the dense ones at every size. The gap is larger for the small models and closes slowly as the dense model gets big. The authors read this as evidence that experts help a model “handle heterogeneous data more effectively”: they have room to specialize, while a small dense model must compromise.

The hero’s fitted lines show the same thing. At 1018 FLOPs the sparse line sits 0.14 below early fusion, at 1022 about 0.065, at 1024 about 0.04. (These gaps come from our arithmetic on the paper’s two fitted lines, not from a number printed in the paper.)

A different best split

Fitting the Chapter 1 formula to the sparse runs, with N as active parameters, gives the Table 2 row for sparse early fusion: α = 0.710, β = 0.372, a = 0.361 and b = 0.656. Two things stand out.

First, α is far larger than for dense models (0.301). The paper explains that the sparse model holds many more total parameters than the active count shows, so each extra active parameter is backed by more stored weights. Second, b is much larger than a. Run the Chapter 3 rule: double the budget and the best active size grows 20.361 = 1.28 times, while the best token count grows 20.656 = 1.58 times. For a dense model the two factors were 1.44 and 1.39. So for sparse models the paper’s advice flips: scale tokens faster than active parameters. That matches Figure 1, where the parameters-per-token ratio of the sparse model falls with compute.

The loss line itself is close to the dense one: exponent −0.048 against −0.049, with a smaller constant, 27.086 against 29.574. The paper adds that sparse models take longer to reach saturation. The authors also tried a scaling law that includes the sparsity as a variable (Appendix C.7) and report consistent conclusions.

Learned routing beats a fixed rule

Figure 11 compares the two routing styles of Chapter 6 across sizes. The learned, modality-agnostic router has the lower loss on both image-caption and interleaved data. The fixed image-FFN and text-FFN split is worse. The model does better when it chooses its own division of labour.

Does it help on real tasks?

Everything so far is validation loss. To check that loss tracks usefulness, the authors fine-tune on the LLaVA instruction mixture and score eight tests (Table 4). All three models are native, at 1.5B scale, and pre-trained on 300B tokens.

ModelAverageTextVQAGQACOCO (CIDEr)TextCaps (CIDEr)
Late fusion46.825.865.870.750.9
Early fusion47.628.165.472.053.8
Early fusion with experts48.230.065.469.655.7

Table 4, four of the eight columns. TextVQA and GQA are question-answering tests on images; CIDEr scores how closely a written caption matches human captions, higher is better. The other columns (VQAv2, OKVQA, VizWiz) are in the paper.

The averages keep the order the loss predicted: late fusion 46.8, early fusion 47.6, experts 48.2. But read the cells. Late fusion wins GQA. The expert model is lowest on COCO captioning. The paper says Table 4 “confirms the ranking”, but the gaps are small, the models are small and the finetuning set is small. The scores sit well below the state of the art, as the authors note.

Did the experts specialize?

To measure it, the authors take each expert in a layer and look at the tokens it received. Let p be the share of those tokens that are text. If p is near 0 or 1, the expert is a specialist. If p is near 0.5, it is a generalist. The binary entropy H(p) = −p log2 p − (1 − p) log2 (1 − p) measures the mixing: 1 bit at p = 0.5 and 0 bits at 0 or 1. The specialization score of a layer is the average over its experts of 1 − H(p). High means specialists; low means shared.

Score one expert

The curve is the score 1 − H(p) as the text share p changes. The bar below it shows what the expert receives. Drag the share, or press a preset.

The formula is the paper’s (Section 4.4, with binary entropy in bits). The presets are illustrative points on the curve, not measured experts. The layer score averages this value over all eight experts.

Take an expert that receives 90% text. Then H(0.9) = −0.9 × (−0.152) − 0.1 × (−3.322) = 0.137 + 0.332 = 0.469 bits, and the score is 1 − 0.469 = 0.53. An expert at 50/50 scores 0. An expert at 99% scores 0.92.

What did they find? Figure 12 plots the score by layer: it is highest in the earliest layers, falls through the middle and rises again in the last layers. Figure 13 shows the load per expert: the first layer has the most single-modality experts. The paper’s reading is that middle layers hold higher-level features that can be shared across modalities, and cites work showing that alignment between modalities grows with depth. A plausible gloss, ours and not tested, is that early layers face raw patches and raw words, which look very different, and the last layers must write text. Alongside the specialists, many experts are shared between text and image, which a fixed split cannot give. All of this is from an early-fusion MoE with 1B active parameters trained for 300B tokens.

Symptom, measurement, fix. A reader might fear that specialization is an artifact of the data: the HQITP data has far more image tokens than text tokens, so experts could seem specialized just from imbalance. The authors plot the score on both HQITP and Obelics. HQITP gives higher scores, as expected, but both show the same up-down-up trend across layers. Even so, the paper does not run a test that removes the imbalance, so treat the trend as strong evidence, not proof.

An expert in the paper’s MoE receives 50% image tokens and 50% text tokens. What is its specialization score, 1 − H(p)?

Chapter 8

Field Guide: The Paper on One Page

Review the argument, keep the numbers and terms, run the core as code, and see the limits and where to read next

You can now rebuild the paper’s argument from memory. Models that see and read can be built two ways. The authors measured both with a scaling law fitted to 457 trained models. The law says the loss is a floor plus a size gap plus a data gap, and the best use of a budget follows from its exponents. They found that early fusion matches late fusion in loss, costs less to train and to serve, and that experts push the loss lower still.

The numbers and terms

WhatNumberWhy it matters
Models trained457; 0.3B to 4B active parameters; 250M to 400B tokensThe size of the evidence behind the fits
Image to tokens224 px, 14 × 14 patches: 256 tokens of 588 numbersOne linear layer is the only image-specific part of early fusion
The decoder24 layers; width 800 (275M) to 3,232 (3.35B)Depth fixed, width grows with size
ComputeC = 6ND (early); 6(NvDv + ND) (late)Turns a budget into a model-size and token trade
Loss law, early fusionE 1.904, α 0.301, β 0.335Ten times the parameters halves the size gap
Loss against compute29.574 C−0.0492 early; 29.923 C−0.0494 late; 27.086 C−0.048 sparseEarly and late fusion coincide; experts sit lower
Best splitN* ∝ C0.526, D* ∝ C0.473 (text: 0.46 and 0.54)Grow parameters and tokens about equally
Parameters per token, N/DC0.053 early; C0.076 late; C−0.312 sparseLate fusion needs a bigger network to serve
Default mixture45% captions, 45% interleaved, 10% textMore image tokens shift the split toward tokens
Scratch against warm startUnder 100B tokens on captions; up to 1T on interleaved and textNative training catches up with a warm start
Sparse setup8 experts, top-1, balance weight 0.01, droplessSame compute per token, more stored weights
Sparse lawα 0.710, β 0.372, a 0.361, b 0.656Scale tokens faster than active parameters
Finetuned average (Table 4)Late 46.8, early 47.6, experts 48.2The loss ranking carries over, with small gaps
Fit spread (100 resamples)Standard deviation: α 0.101, β 0.029Token exponent is better pinned than size exponent

The core as code

This is a compact version of each mechanism: patchify an image, fit the loss law, get the best split, and route tokens with a balance loss. It is our sketch of the paper’s method, not the authors’ released code.

pythonimport numpy as np
from scipy.optimize import minimize
from scipy.special import logsumexp, softmax

def embed(img, text_ids, W_patch, E_text, p=14):
    # img (224, 224, 3) -> 256 patches of 588 numbers -> one linear layer
    h = img.shape[0] // p
    x = img.reshape(h, p, h, p, 3).transpose(0, 2, 1, 3, 4).reshape(h * h, p * p * 3)  # (256, 588)
    return np.concatenate([x @ W_patch, E_text[text_ids]])        # (256 + T, d)

def attn_mask(n_img, n_txt):
    n = n_img + n_txt; m = np.zeros((n, n), bool)
    m[:n_img, :n_img] = True                                       # image: bidirectional
    for t in range(n_txt): m[n_img + t, :n_img + t + 1] = True     # text: all images, earlier text
    return m

def fit_loss_law(N, D, L, delta=1e-3):                          # L = E + A/N^al + B/D^be
    def obj(th):
        a, b, e, al, be = th
        pred = logsumexp([a - al * np.log(N), b - be * np.log(D), e + 0 * np.log(N)], axis=0)
        r = pred - np.log(L)
        return np.where(abs(r) <= delta, .5 * r**2, delta * (abs(r) - .5 * delta)).sum()  # Huber
    best = None
    for a0 in range(0, 31, 5):                                     # grid of starting points
        for al0 in (0, .5, 2.5):
            res = minimize(obj, [a0, a0, 0.5, al0, al0], method="L-BFGS-B")
            if best is None or res.fun < best.fun: best = res
    a, b, e, al, be = best.x
    return np.exp(e), np.exp(a), np.exp(b), al, be                 # E, A, B, alpha, beta

def compute_optimal(C, A, B, al, be):                           # minimize L(N) with D = C / 6N
    N = (al * A / (be * B)) ** (1 / (al + be)) * (C / 6) ** (be / (al + be))
    return N, C / (6 * N)                                          # N* ~ C^(be/(al+be))

def route_top1(x, W_r, E=8):                                   # x (T, d), W_r (d, 8)
    logits = x @ W_r                                               # (T, 8)
    idx = logits.argmax(1)                                         # (T,) one expert per token
    f = np.bincount(idx, minlength=E) / len(idx)                   # share of tokens per expert
    balance = E * (f * softmax(logits, 1).mean(0)).sum()           # 1.0 when even (weight 0.01)
    return idx, balance

A checklist for reading a scaling-law claim

The limits the authors state

Where it sits, and where to read next

The paper is a reading for Lecture 8, Large Multimodal Models, which shows the glued designs it tests. The loss-law and compute-optimal ideas come from Chinchilla, and CS336 scaling laws builds them from scratch. The expert layer is taught in Mixture of Experts. For the building blocks, see Vision Transformer (the patch idea), Contrastive CLIP (the late-fusion encoder’s family), Vision-Language Models and Lecture 5, Fusion (the fusion vocabulary this paper uses at scale).

The takeaway. The question “glue or grow” has a measured answer: no inherent advantage to glue. One transformer with a single linear patch layer reaches the same loss, with fewer parameters, less memory and less training time, and experts that sort themselves by modality push it lower. The method that made it checkable is the loss law: a floor, two shrinking gaps, and a closed-form split of any budget.

Credit

Paper: Mustafa Shukor, Enrico Fini, Victor Guilherme Turrisi da Costa, Matthieu Cord, Joshua Susskind and Alaaeldin El-Nouby, “Scaling Laws for Native Multimodal Models” (Apple and Sorbonne University, 2025), arXiv:2504.07951. All figures, tables and numbers here are theirs unless a line says illustrative; the explanations and devices are ours. Assigned in MIT 6.S985 Multimodal AI, Spring 2026 (Paul Liang).

Now press Present or Teach and explain, from memory, why the best model size grows as C to the power β over (α plus β), and why late fusion needs more parameters per token. Then go back to the compute dial at the top, move it to 1024, and say which of the three gaps you can explain.

A team wants a seeing-and-reading model that is cheap to serve. What does this paper’s evidence favour?
Scaling Laws for Native Multimodal Models
Read the paper · Back to Lecture 8, Large Multimodal Models