Chinchilla

A 70 billion parameter model trained on 1.4 trillion tokens beat Gopher, four times its size, for the same compute. Slide the model size and find the best split of one budget yourself.

Learn how DeepMind decided how big a language model should be, and how long it should train, when the compute budget is fixed.

Press Chinchilla, then Gopher, then drag the model size. The budget never changes: a bigger model simply sees fewer tokens. Watch where the loss curve bottoms out. Then we build the answer piece by piece, from counting the cost of one training run to the three ways the authors fitted the curve.

You need a feel for what a number of parameters is and a little algebra. We build everything else from zero: tokens, FLOPs, loss, power laws.

Paper: Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Laurent Sifre and colleagues at DeepMind, “Training Compute-Optimal Large Language Models”, arXiv 2203.15556 (2022). A reading for the Large Multimodal Models lecture of MIT MMAI, Multimodal AI (Spring 2026). Every idea is re-taught here in our own words and devices.

Split the budget

70B

The budget is 5.76 × 1023 FLOPs, the paper’s number for Gopher (Figure 2). Tokens come from C = 6ND. The loss curve is the paper’s fitted formula (Equation 10) with its printed constants, so it is a model of the 400 runs, not a measurement. Accuracy figures are the paper’s Table 6 (MMLU, 5-shot).

Chapter 0

One Budget, Two Dials

See why a fixed pile of compute forces a choice between a bigger model and more reading, and why the field chose wrong for two years

Imagine you have rented a large room of computer chips for three months. The rent is paid. You cannot add more chips, and you cannot stay longer. In that time you must train one language model, and you only get one try, because a second try costs another three months.

Before you press start, you must make two choices. How big should the model be? And how much text should it read while it learns? Each choice eats the same fixed budget. A bigger model costs more for every word it reads, so it can read fewer words. A smaller model reads more words in the same time.

This paper, Training Compute-Optimal Large Language Models from DeepMind, asks exactly that: given a fixed budget, what is the best split? It is a reading for the lecture on large multimodal models, because every big model in that lecture, text or image or video, was sized by a rule like this one. The rule it replaced was wrong, and the paper shows how.

Four words you need

A parameter is one number inside the model that training adjusts. Think of the knobs on a huge mixing board. A model with 70 billion parameters has 70 billion knobs. We write the count as N. The count tells you how much the model can remember and how much arithmetic each word costs.

A token is one small piece of text, often a word or part of a word, that the model reads as a single unit. The model never sees letters. It sees a stream of tokens. We write the number of tokens used in training as D, for data. Reading more tokens is like reading more books.

A FLOP is one floating point operation, one addition or one multiplication of two decimal numbers. The total number of FLOPs a training run needs is the compute, and we write it as C. Chips are rented by how many FLOPs they can do, so C is the budget. Gopher’s training run used about 5.76 × 1023 FLOPs, which is the number you will see on every picture in this lesson.

The loss, written L, is the score for how badly the model predicts the next token. Lower is better. It is the number training tries to push down, and it is what the paper uses to compare choices.

The habit that needed a check

In 2020 a paper by Kaplan and colleagues showed that a model’s loss falls smoothly as it grows. The field read this as a message: make models bigger. Table 1 of the paper shows the result. GPT-3 had 175 billion parameters and saw 300 billion tokens. Jurassic had 178 billion and 300 billion. Gopher had 280 billion and 300 billion. MT-NLG had 530 billion and 270 billion.

Look at the token column. Model size jumped from 175 billion to 530 billion, yet the text it read stayed near 300 billion. The field turned only one dial. The authors suspected that this wasted money: the models were too big for what they had read.

To test the idea they trained over 400 models, from under 70 million to over 16 billion parameters, on 5 to 500 billion tokens. Then they trained one big model of their own to check the result. It is called Chinchilla. It has 70 billion parameters, four times fewer than Gopher, and it read 1.4 trillion tokens, about about 4.7 times more. The budget is the same. On the MMLU exam benchmark it scored 67.6%, against 60.0% for Gopher.

The budget as a rectangle

Here is a picture of the trade. The budget is a rectangle. Its width is the model size N. Its height is the number of tokens D. Chapter 1 shows that the compute is proportional to the area, N times D. Because the budget is fixed, the area never changes. Drag the width and watch the height give way.

Reshape the budget

The shaded rectangle always has the same area. Drag the model size. The outlines mark Gopher (wide and short) and Chinchilla (narrow and tall).

70B

The area is the compute budget C = 6ND (Chapter 1) fixed at 5.76 × 1023 FLOPs. The outlines use the models’ real sizes (280B and 70B) and tokens from the same formula, so Gopher’s outline shows 343B tokens and not its real 300B. Table 1 of the paper lists 300B.

Two things stand out. First, no split is free: more width means less height, always. Second, the corners are bad. A very wide, very short rectangle is a giant model that has barely read anything. A very tall, very thin one is a tiny model that has read the whole library but cannot hold it. The best split sits somewhere in the middle, and the rest of the paper is a hunt for that middle.

Why it matters beyond one model

A model that is too big is expensive twice. You pay too much to train it, and you pay again every time someone uses it, because each answer needs arithmetic for every parameter. Chinchilla is four times smaller than Gopher, so every answer it gives costs about a quarter of the arithmetic. That is why the paper stresses that its method helps with inference (running the model) as well as training.

The paper finds its answer three different ways, and all three agree. You will build each one. By the end you will be able to take any budget and say how big the model should be, how many tokens it should read, and why.

  1. Counting the cost: why compute is six times parameters times tokens.
  2. What came before: Kaplan’s recipe, and the fixed learning-rate schedule that fooled it.
  3. Approach 1: train many sizes, take the lower envelope.
  4. Approach 2: hold the budget fixed, find the valley.
  5. Approach 3: write the loss as a formula and solve it.
  6. The answer: grow model and data in equal proportion, about 20 tokens per parameter.
  7. Chinchilla: the test run, its shapes and its changes.
  8. Results and limits: the tables, the ablations, the cautions.
  9. Field guide: numbers, checklist, code.
Why can a fixed compute budget not buy both a huge model and a huge amount of training text?

Chapter 1

Counting the Cost: C = 6ND

Count the arithmetic in one training step, and see why compute equals six times parameters times tokens

You are the person paying for the chips. A friend says, “Training this model will take 5.76 × 1023 operations.” You want to check that number yourself, with a pencil, before you pay. Where does such a number come from?

The answer is one short formula, and the whole paper stands on it. Every plot, table and fit in the paper turns a budget C into a model size N and a token count D, so the link between the three must be exact enough to trust. This chapter derives it from one layer of arithmetic.

The smallest case: one layer, one token

A language model is a stack of layers. Most of the work in a layer is a matrix multiplication: a row of numbers (the token, as a list of 4,096 numbers) is multiplied by a grid of weights. Each weight is a parameter. Say the grid has 4,096 rows and 4,096 columns. That is 4,096 × 4,096 = 16,777,216 weights, about 16.8 million parameters.

How much arithmetic does one token need? Each output number is a sum: take 4,096 inputs, multiply each by one weight, then add the results. That is 4,096 multiplications and about 4,096 additions. There are 4,096 outputs. So the layer does about 4,096 × 4,096 multiplications and the same number of additions.

Count it per weight. Every weight takes part in exactly one multiplication and one addition, for one token. So the cost is 2 FLOPs per parameter per token. This is the forward pass, the trip from input to prediction. For a whole model of N parameters, the forward pass costs about 2N FLOPs per token.

Going backward costs twice as much

Training needs more than a prediction. After the model predicts, we measure the loss and ask: how should each weight change to lower it? The backward pass answers that. It sends blame back through the layers, one layer at a time, using the chain rule.

At each layer the backward pass does two matrix multiplications of the same size as the forward one. The first finds how the loss depends on the layer’s input, so the blame can keep travelling backward. The second finds how the loss depends on the layer’s weights, so the weights can be updated. Each costs 2 FLOPs per weight, so together they cost 4 per weight.

The paper states this as an assumption borrowed from Kaplan: the backward pass has twice the FLOPs of the forward pass. Add them up: 2 forward and 4 backward is 6 FLOPs per parameter per token. Multiply by the number of tokens and you have the formula.

C
The training compute, in FLOPs. This is the budget you pay for.
6
2 FLOPs forward plus 4 FLOPs backward, for each parameter and each token.
N
The number of parameters in the model, counting the embedding table (the lookup grid that turns each token into a list of numbers, Chapter 7).
D
The number of tokens the model trains on. One pass over the data, so no token counts twice.

The paper writes this as FLOPs(N, D) ≈ 6ND in Section 3.3 and cites Kaplan. In Appendix F the authors do the exact count instead. They add up the embeddings, the attention scores, the softmax and every feed-forward block. They compare it with 6ND in Table A4 and call the differences very small, so the simple form is safe for planning.

Check it on real models

Take Chinchilla: N = 70 billion, D = 1.4 trillion. Then 6 × 70 × 109 × 1.4 × 1012 = 5.88 × 1023. The paper’s exact count for the budget is 5.76 × 1023, so the shortcut is 2% high.

Now Gopher: N = 280 billion, D = 300 billion. Then 6ND = 5.04 × 1023, which is 12% below 5.76 × 1023. The gap is larger here because the paper’s exact count (Appendix F) also adds the attention scores and the output layer, which 6ND leaves out. The point stands: the two models cost about the same, and that equality is the whole design of the experiment. Pick a model below and see the cost split into its forward and backward parts.

Price a training run

Pick a model from the paper’s Table 1. The top bar is its compute 6ND, split into the forward pass (blue-grey) and the backward pass. The bottom bar is how many tokens it read for each parameter.

Sizes and token counts are the paper’s Table 1. Compute is the 6ND shortcut, so it can differ from the paper’s exact count by about ten percent (Gopher: 5.04 × 1023 here against 5.76 × 1023 counted). Tokens per parameter is simply D divided by N.

Look at the second bar. Every large model before Chinchilla read between half a token and under two tokens for each parameter. Chinchilla read twenty. The budget is not what changed. How it was spent changed.

The shortcut as code

The formula needs one line. The tiny check below counts the real multiplications in one matrix product and confirms the 2 FLOPs per weight, so you do not have to take the six on trust.

pythonimport numpy as np

d = 4096
x = np.random.randn(1, d)          # one token: shape [1, 4096]
W = np.random.randn(d, d)          # weights:   shape [4096, 4096]  (16.8M parameters)

fwd_flops = 2 * d * d              # d*d multiplies + d*d adds = 2 per weight
bwd_flops = 2 * fwd_flops          # dX = dY @ W.T  and  dW = x.T @ dY
print(fwd_flops + bwd_flops == 6 * W.size)   # True: 6 per parameter per token

def train_flops(N, D):
    return 6 * N * D                # the paper's shortcut (Section 3.3)

print(train_flops(70e9, 1.4e12))     # 5.88e+23, vs 5.76e+23 counted exactly
A model has twice as many parameters and trains on the same number of tokens. What happens to the compute 6ND?

Chapter 2

What Came Before: Kaplan and the Fixed Schedule

Read the earlier recipe, see the one training detail that tilted it toward bigger models, and learn why the paper fixes it

It is 2020. You run a research team and you need to decide how big to make your next model. A new paper by Kaplan and colleagues gives you a clean rule. It says the loss falls in a smooth, predictable way as the model grows, and it says how to spend more compute when you get it.

You follow the rule. Your models get bigger and bigger, and each one is better than the last. Nothing seems wrong. Yet two years later this paper shows that the rule sent the whole field to the wrong place. To see how, we first need to read the rule closely.

A power law, in plain words

A power law is a relationship of the form y = k · xa. The number a is the exponent. If a is 0.5, then y is the square root of x scaled by k. Doubling x multiplies y by 20.5, about 1.41.

Power laws have a handy picture. Take the logarithm of both sides: log y = log k + a · log x. That is a straight line, and a is its slope. So on a plot where both axes are logarithmic, a power law is a straight line, and you can read its exponent from how steep the line is. The paper uses such plots everywhere (the axes of Figure 1 run from 1017 to 1025 FLOPs).

Kaplan found that the loss follows a power law in the number of parameters. The paper also says Kaplan concluded that large models should not be trained to their lowest possible loss. Both authors agree on that. They disagree on how to split extra compute.

Two rules, side by side

The paper states Kaplan’s advice in one sentence: for a 10 times larger compute budget, the model should grow by 5.5 times, and the tokens by only 1.8 times. Check it: 5.5 × 1.8 = 9.9, close to 10, so the compute works out (remember C ≈ 6ND).

Turn those multipliers into exponents. If N grows as Ca, then 10a = 5.5, so a = log10(5.5) = 0.74. For tokens, 10b = 1.8 gives b = 0.26. The last row of the paper’s Table 2 prints Kaplan’s exponents as 0.73 and 0.27, which matches. So Kaplan’s rule is: put about three quarters of any new compute into model size and only one quarter into data.

The paper’s rule is different. For a 10 times larger budget, both N and D should grow by √10 = 3.16 times. In exponents, a = 0.5 and b = 0.5. The model and the data grow in step. Both rules obey a + b = 1, because their product must equal the budget. They just split the growth differently.

Nopt
The best model size for a budget C. “Best” means the lowest final loss.
Ca
The budget raised to the power a. Kaplan: a is about 0.73. This paper: a is about 0.5.
Dopt
The best number of training tokens for the same budget.
Cb
The budget raised to the power b. Kaplan: b is about 0.27. This paper: b is about 0.5.

The learning rate, and a schedule that never finished

Why did the two groups disagree? The authors point to one detail of how the training runs were set up. To follow it, you need one idea from training.

The learning rate is the size of each step the model takes when it updates its weights. Early on, big steps help. Late in training, big steps make the weights jitter around the best spot and never settle. So training uses a learning-rate schedule: a plan for shrinking the steps. A popular plan is the cosine schedule, which follows half a cosine wave from the starting rate down to a small final rate. In this paper the rate falls by a factor of 10 over the schedule.

Here is the catch. The loss only becomes good once the learning rate has come down. A run that is stopped while the rate is still high reports a worse loss than it would have reached with a finished schedule.

The paper says Kaplan used one fixed number of training tokens and one fixed schedule for all models. Suppose the schedule was planned for 130 billion tokens. You then read the loss of a model after only 40 billion tokens, to ask what a smaller training run would give. At that point the learning rate is still high, so the loss looks bad. The small-data option appears worse than it really is. Drag the stop point below and see how high the rate still is.

Stop a run early

The curve is the learning rate over training. The vertical line is where you stop reading the loss. Switch between a schedule planned for 130B tokens and one planned for exactly where you stop.

40B

The 130B horizon and the ten-fold decay are the paper’s (Section 2, Appendix D.1). The cosine formula, rate = peak × (0.1 + 0.9 × ½(1 + cos(π t / T))), is one standard way to write a ten-fold cosine decay. The paper reports performance “noticeably degraded” when the cycle overshoots the training length by more than 25%, so the device flags that case.

Read the chip while you drag. With the fixed schedule, a stop at 40B tokens leaves the rate near 81% of its peak. With the matched schedule the same stop is at the 10% floor, where it belongs. Every short run in the fixed-schedule world looked worse than it was, so bigger models trained on little data looked unbeatable.

That is the failure in three beats. The symptom: models trained on fewer tokens looked poor, so the data-light, model-heavy choice kept winning. The measurement: the paper reruns the comparison with the cosine length matched to the tokens and shows, in Figure A1, that overshooting the schedule by more than 25% hurts. The fix: for every run, set the schedule length to match its training tokens. Everything in the next chapters uses that fix.

The authors name a second difference. Most of Kaplan’s runs were small, many under 100 million parameters. This paper trains most of its models at over 500 million, up to 16 billion, because the frontier of best loss per FLOP curves slightly (Appendix E). Small models alone can bend the line.

Why did a fixed 130B-token schedule make short training runs look worse than they really were?

Chapter 3

Approach 1: The Training-Curve Envelope

Train many model sizes, watch every loss curve, and keep the lowest one at each budget

You have five racing cars of different sizes and a track marked in kilometres of fuel. You run all five. The small car is quick at first but runs out of speed. The big car is slow off the line but has a higher top speed. You want to know: for each amount of fuel, which car is in front?

That is the first of the paper’s three methods. Replace cars with models, fuel with FLOPs, and speed with low loss. You do not need a theory. You only need to run enough models and look at who wins at each budget.

The recipe, step by step

The authors train a family of models, from 70 million to over 10 billion parameters. Each size is trained four times, with four different training lengths that span a factor of 16. For each run, the learning rate decays by a factor of 10 on a cosine schedule matched to that run’s own length. Chapter 2 explained why that matching matters.

Each run produces a training-loss curve: a loss value at every step. The authors smooth it with a short sliding average (a Gaussian window of 10 steps) so that noise does not fool them. Then they convert the x-axis from steps into FLOPs, using C ≈ 6ND. Now every run is a curve of loss against compute.

Next, they interpolate each curve so that it gives a loss for any FLOP count. At each of 1,500 FLOP values, spaced evenly on a log scale, they ask: which run has the lowest loss here? That run’s model size is the best N for that budget, and the number of tokens it has read is the best D. The line that follows the lowest points is the envelope of the training curves. Figure 2 of the paper (left) shows that envelope.

One detail deserves a mention. The authors note that every chosen point lies within the last 15% of its run. That fits the schedule rule: a run only reports its best loss once its learning rate has come down.

From the envelope to a rule

The envelope gives a long list of pairs: at this budget, this model size and this many tokens. To turn the list into a rule, the authors fit a power law to each. They write Nopt ∝ Ca and Dopt ∝ Cb. Because the two lines are straight on log-log paper, the fit is a plain line fit.

Try it by hand with two entries from the paper’s Table 3. A 1 billion parameter model is optimal at 1.21 × 1020 FLOPs. A 10 billion parameter model is optimal at 1.23 × 1022 FLOPs. The budget grew by a factor of 1.23 × 1022 / 1.21 × 1020 = 101.7. The model grew by a factor of 10. So 101.7a = 10, which means a = log(10) / log(101.7) = 1 / 2.007 = 0.498. That is the paper’s a = 0.50.

The same trick works for tokens. The 1B model reads 20.2 billion tokens and the 10B model reads 205.1 billion. The ratio is 10.2, so b = log(10.2) / log(101.7) = 0.50 as well. The tokens grew by the same factor as the model, which is just what “grow them equally” means.

Nopt ∝ C0.50
Approach 1’s result for model size. A budget 100 times larger wants a model 10 times larger, since 1000.5 = 10.
Dopt ∝ C0.50
Approach 1’s result for tokens. The same 100 times larger budget wants 10 times more tokens.

Table 2 of the paper reports a = 0.50 with a range of 0.488 to 0.502, and b = 0.50 with a range of 0.501 to 0.512. These ranges are the 10th and 90th percentiles from a bootstrap, a way of rerunning the fit on 100 random 80% samples of the data to see how much the answer moves. It barely moves.

See the envelope form

Real training curves are noisy and need hardware. The device below draws a stand-in for them, using the paper’s fitted loss formula (Chapter 5 derives it), so you can see the shape of the argument. Each curve is one model size. Drag the budget and see which size is in front.

Find the lowest curve

Each thin line is one model size, trained on more and more tokens. The thick line is the envelope, the lowest loss at each budget. Drag the budget to see which model sits on it.

2.0 × 10²⁰

Illustrative. These curves come from the paper’s fitted formula (Equation 10) with D = C / 6N, not from the 400 real runs, and they ignore the learning-rate schedule. The real curves in Figure 2 are noisy, and the five sizes here (75M, 250M, 1B, 2.5B and 10B) are a subset of the sizes the paper trained.

Notice what the envelope shows. Each small model wins early and then flattens: it has run out of room. Each large model starts worse, because it wastes compute on weights it has not trained yet, and then wins later. The winner changes as the budget grows. The paper’s Figure 2 (centre and right) plots the winning N and D against C, and both are straight lines. At Gopher’s budget of 5.76 × 1023 FLOPs, those lines point to a model of about 67 billion parameters and 1.5 trillion tokens. That is Table 3’s middle row, and it is where Chinchilla comes from.

What this method cannot see

The envelope can only choose among sizes that were trained. Between two trained sizes it interpolates. It also depends on the smoothing and on how the learning-rate schedule is matched. The other two approaches check it from different directions, and the next chapters show them.

In Approach 1, how is the best model size for a given budget found?

Chapter 4

Approach 2: IsoFLOP Valleys

Hold the budget fixed, vary only the model size, and read the best size from the bottom of a valley

Approach 1 followed every run for its whole life and picked winners along the way. It answers the question, but in a roundabout way. Here is a more direct one. Give every runner exactly the same fuel. Change only the size of the car. Then ask who finished best.

That is Approach 2. The word IsoFLOP means “equal FLOPs”. An IsoFLOP profile is a plot where every point used the same budget C, and only the model size changes. The question “for this budget, what is the best parameter count?” is answered by looking at one curve.

How the experiment works

The authors choose nine budgets, from 6 × 1018 to 3 × 1021 FLOPs. For each budget they train a range of model sizes, up to 16 billion parameters. Because the budget is fixed, the token count follows from the size: D = C / (6N). A small model reads many tokens. A big model reads few.

Every run uses a cosine schedule matched to its own length, as Chapter 2 required. They record only the final training loss of each run. This is the difference from Approach 1, which used points all along each curve. Here each run gives one number.

Plot final loss against model size, once per budget, and a clear shape appears: a valley. In Figure 3 (left) the authors say they made sure to train enough different sizes to see a clear minimum in every case.

Why there is a valley

Two forces pull in opposite directions. On the left side, the model is small. It reads plenty of text, but it has too few parameters to store what it reads, so its loss is high. On the right side, the model is large but has read only a thin slice of text, because the budget ran out. Its loss is high because most of its parameters were never trained properly. The best size lies between the two, where neither force dominates.

The picture should look like a bowl. The paper fits a parabola to each bowl to find where the bottom sits.

Finding the bottom with a parabola

A parabola is the shape y = p·x² + q·x + r. Its lowest point, when p is positive, is at x = −q / (2p). We fit it to the points and read the bottom from the formula. We use x = log10(N), because model sizes span many powers of ten and a valley on a log axis is much more symmetric.

Work one example. Take the budget C = 1021 FLOPs, and use the paper’s fitted loss formula (Chapter 5) to stand in for measured runs. Three trial sizes give three losses:

  1. N = 1 billion, so D = 1021 / (6 × 109) = 167 billion tokens. Loss = 2.340.
  2. N = 4 billion, so D = 42 billion tokens. Loss = 2.347.
  3. N = 16 billion, so D = 10 billion tokens. Loss = 2.471.

With x = 9, 9.602 and 10.204, the three points give p = 0.161 and q = −2.98 in the form above. The vertex is at x = 2.98 / (2 × 0.161) = 9.26, which is N = 109.26 = 1.83 billion parameters. The best of the three trial sizes was 1B, but the true bottom is between 1B and 4B, and the parabola finds it. That is the whole point of fitting: you read the minimum between your samples.

x★
The bottom of the valley, measured as the base-10 logarithm of the model size.
q
The linear coefficient of the fitted parabola in x = log10(N). Negative when the valley bottom is at positive x.
p
The quadratic coefficient. It is positive for a bowl, and larger p means a narrower bowl.
Nopt
The best model size for this budget. The matching token count is D = C / (6 Nopt).

Now move the budget in the device below. Each dot is a trial model. The purple line is the parabola fitted to the dots, and the marker is its bottom. The fainter markers are the bottoms for the smaller budgets, so you can watch the valley slide to the right.

Slide the valley

Pick one of the paper’s nine budgets. The dots are trial model sizes at that budget, and the line is a parabola fitted to them.

1 × 10²¹

The nine budgets (6e18, 1e19, 3e19, 6e19, 1e20, 3e20, 6e20, 1e21, 3e21) are the paper’s. The loss values are illustrative: they come from the fitted formula (Equation 10), with seven trial sizes spaced a factor of 2.2 apart and centred on that formula’s own minimum. The paper’s dots are measured, and its parabolas are fitted the same way.

After fitting nine parabolas, the authors have nine pairs (C, Nopt). They fit a power law through them in the same way as in Approach 1. The result is Nopt ∝ C0.49 and Dopt ∝ C0.51. Table 2 gives the ranges: 0.462 to 0.534 for a, and 0.483 to 0.529 for b. This is the same answer as Approach 1, from a different set of numbers. Figure 3 (centre and right) shows the lines, and at Gopher’s budget they point to about 63 billion parameters and 1.4 trillion tokens.

Why two methods agreeing matters

Approach 1 uses the whole of each curve and many points per run. Approach 2 uses the end of each run and a hand-picked set of sizes. Their errors are different. When two quite different procedures give a = 0.50 and a = 0.49, a reader should trust the answer more than either alone.

Appendix C of the paper adds a third check of a different kind: it repeats Approach 2 on two other datasets, C4 and GitHub code, and finds the same equal scaling. The authors conclude that the result does not depend on the dataset, as long as training runs for less than one epoch.

In an IsoFLOP profile, what stays the same across the points, and what changes?

Chapter 5

Approach 3: A Formula for the Loss

Write the loss as three simple terms, fit the constants once, and solve for the best split with algebra

The first two approaches pick winners from runs. A third idea skips the picking. Suppose you could write one formula that predicts the loss for any model size N and any token count D. Then you would not need to train anything new. You would just ask the formula where the loss is lowest for a given budget.

This chapter builds that formula from three plain observations, shows how the authors fit it, and then solves it by hand. The algebra is short, and the answer is one of the nicest results in the paper.

Three reasons a model gets things wrong

Take a language model that predicts the next token. It will still make mistakes. The paper lists three reasons, and each becomes a term in the formula.

The first reason is that text is not fully predictable. After “I bought a loaf of”, many words could follow. Even a perfect predictor would be uncertain. The loss of that perfect predictor is the entropy of natural text, written E. No amount of model size or data can push the loss below E. The fit puts E at 1.69.

The second reason is that the model is too small. A model with N parameters is a limited kind of function, and even the best one of that size misses things a perfect predictor would get right. Bigger N shrinks this gap, but with fading returns. The term is A / Nα.

The third reason is that training is short. The model has seen D tokens, only once each, and has taken a limited number of steps. It has not converged. More data shrinks this gap too, also with fading returns. The term is B / Dβ.

L(N, D)
The predicted final loss for a model with N parameters trained on D tokens. Lower is better.
E
The loss floor, the entropy of natural text. The paper’s fit: 1.69.
A / Nα
The penalty for a model of limited size. The fit: A = 406.4, α = 0.34.
B / Dβ
The penalty for limited data and training. The fit: B = 410.7, β = 0.28.

This is Equation 2 of the paper, with the fitted values from Appendix D. The authors derive its shape from a standard split of risk into the best possible loss, the loss from using a model class of size N, and the loss from training on a finite sample. They argue the model term should fall like a power of N and the data term like a power of D, and then they let the data choose the numbers.

Fitting the five constants

The formula has five unknowns: E, A, B, α and β. The authors have losses from all their runs in Approaches 1 and 2. They search for the five numbers that make the formula's predictions closest to the observed losses. Two details matter.

First, they compare logarithms of the losses, not the raw losses. A mistake of 0.1 means more when the loss is 2 than when it is 5, and the log handles that. Second, they use the Huber loss for the gaps. The Huber loss treats small gaps like squared error and large gaps like absolute error, so a few wild points cannot drag the fit. They set its threshold δ to 10−3. A larger δ made the fit chase the small-compute runs and predict the large runs badly.

They minimise this with L-BFGS, a standard optimiser, started from a grid of different starting points to avoid bad local minima. The best starting point is not on the edge of the grid, which tells them the search did not get stuck at a boundary. The paper also notes that runs with 1021 FLOPs or less have larger errors, and that the Huber loss quietly treats them as outliers.

Solving for the best split

Now the payoff. We want the N and D that minimise L(N, D) when 6ND = C. Replace D by C / (6N), so the loss depends on N alone:

L(N) = E + A · N−α + B · (C/6)−β · Nβ

The first term pulls N up. The last term pulls N down. At the best N the slope is zero. Take the derivative and set it to zero:

  1. −αA · N−α−1 + βB · (C/6)−β · Nβ−1 = 0.
  2. Move one term across: αA · N−α−1 = βB · (C/6)−β · Nβ−1.
  3. Multiply both sides by Nα+1: αA = βB · (C/6)−β · Nα+β.
  4. Solve for N: Nα+β = (αA / βB) · (C/6)β.

Take the root with exponent 1/(α+β):

Nopt
The best model size, a power of the budget, as in the other two approaches.
G
(αA / βB) raised to 1/(α+β). It sets the ratio of model to data. With the fitted constants, G = 1.34.
a
The model-size exponent, β/(α+β) = 0.28 / 0.62 = 0.45. The paper reports 0.46 from unrounded constants.
b
The token exponent, α/(α+β) = 0.34 / 0.62 = 0.55. The paper reports 0.54.

Look at what fell out. The exponents depend only on α and β, the two rates at which size and data help. The constants A and B only shift the ratio through G. And because β (0.28) is a little smaller than α (0.34), data helps a bit more slowly than size, so the optimum asks for slightly faster growth in tokens than in parameters. That is why Approach 3 lands at 0.46 and 0.54, a little away from 0.50 and 0.50.

Take the loss apart

Pick a split of the Gopher budget. The bar is the predicted loss, cut into the floor E, the model term and the data term. Then move α and β and watch the optimum move.

0.34
0.28

E = 1.69, A = 406.4 and B = 410.7 are fixed at the paper’s values (Appendix D.2); α and β start at its 0.34 and 0.28. Gopher uses its real 280B parameters and 300B tokens, Chinchilla 70B and 1.4T. Moving the sliders is a what-if, not a refit: with other exponents the constants A and B would also change.

Run the numbers at the fitted values. Gopher: the model term is 0.052 and the data term is 0.251, so L = 1.69 + 0.052 + 0.251 = 1.994. Chinchilla: model term 0.084, data term 0.163, L = 1.937. Chinchilla gives up a little on the model term and wins much more on the data term. At Gopher’s budget the closed form with these printed constants puts the best size near 32 billion; the paper’s own figure shows 40 billion, because the printed constants are rounded and the answer is sensitive to them. The paper brackets the best size at 40 to 70 billion and tests the upper end.

In L(N, D) = E + A / Nα + B / Dβ, what does the B / Dβ term stand for?

Chapter 6

The Answer: Grow Both Equally

Read Tables 2 and 3, turn three agreeing fits into one rule of thumb, and see how far past the rule the field had gone

You now have three recipes, and each ends in a pair of exponents. Before we use them, line them up. Do they agree, and what do they say in plain terms?

ApproachModel exponent aToken exponent b
1. Minimum over training curves0.500.50
2. IsoFLOP profiles0.490.51
3. Parametric loss fit0.460.54
Kaplan et al. (2020)0.730.27

This is Table 2 of the paper. The three new rows cluster around one half. The old row sits far away. The authors write that all three of their approaches suggest that, as the budget grows, the model size and the amount of data should be increased in approximately equal proportions.

What “equal proportions” means

Put a number on it. Say your budget grows by a factor of 10. With a = b = 0.5, the model grows by 100.5 = 3.16 and the tokens grow by 3.16. Check the product: 3.16 × 3.16 = 10, so the compute works out. With Kaplan’s numbers the model grew 5.5 times and the tokens only 1.8 times.

Equal growth has a surprising consequence. If both grow by the same factor, their ratio never changes. The ratio D / N is the number of tokens per parameter. So the right answer to “how much should it read?” is a roughly constant number of tokens for each parameter, at every scale.

Check the paper’s Table 3 for the value. Divide the tokens by the parameters in each row. For 400 million parameters: 8.0 billion tokens, a ratio of 20.0. For 1 billion: 20.2. For 10 billion: 20.5. For 67 billion: 1.5 trillion, so 22.4. For 175 billion: 3.7 trillion, so 21.1. For 1 trillion: 21.2 trillion, so 21.2. The ratio stays between 20 and 22 across a range of 2,500 times in model size. That is the rule of thumb: about 20 tokens per parameter. The paper does not state this single number. It is our reading of Table 3.

From the rule to a recipe

Write the rule as D = r · N with r near 20. Put it into the cost formula from Chapter 1, C = 6ND. Then C = 6 · r · N², so

N
The model size to train for budget C. It grows as the square root of C, which is the exponent 0.5.
C
Your training budget in FLOPs.
r
Tokens per parameter, about 20 from Table 3.
D
The tokens to train on. With r = 20 it is 20 times the parameter count.
r
The same tokens-per-parameter number, now giving D from N.

C = 5.76 × 1023  →  N = √(5.76 × 1023 / 120) = 6.9 × 1010,   D = 20 × 6.9 × 1010 = 1.4 × 1012

That is 69 billion parameters and 1.4 trillion tokens: Chinchilla, from one line of arithmetic. The authors did not apply this shortcut. They ran their three methods. But the shortcut shows that the answer was a consequence of equal scaling all along.

How far past the line the field had gone

Table 3 lists, for each model size, the compute and tokens that put it on the optimal line. Pick a row below. The solid line shows the tokens each size should read. The dashed line shows what the field actually fed its big models, about 300 billion.

Read Table 3

Choose a model size. The blue line is the tokens Table 3 says it needs. The dashed line is the 300 billion tokens most large models of the time received.

Rows are the paper’s Table 3 (Approach 1): parameters, FLOPs and tokens. Tokens per parameter is our division. Section 3.4 of the paper quotes slightly different figures for the 175B row (4.41 × 1024 FLOPs and over 4.2 trillion tokens), which come from the other approaches.

The dashed line crosses the solid one near 15 billion parameters. Below it, 300 billion tokens is plenty. Above it, the model is starved. A 175 billion model needs 3.7 trillion tokens by Table 3, 12 times more than GPT-3 got. The paper puts the same point in words: a 175 billion parameter model should be trained with a compute budget of 4.41 × 1024 FLOPs and on over 4.2 trillion tokens, and a 280 billion Gopher-like model is optimal at about 1025 FLOPs and 6.8 trillion tokens.

Two more consequences follow. Unless you have a budget of 1026 FLOPs, over 250 times Gopher’s, a model with a trillion parameters is unlikely to be the one to train. And the token counts the table asks for are far beyond what anyone then used, so collecting more and better data becomes as important as building bigger machines. The authors add, as a guess rather than a result, that more data only helps when it is high quality.

A team multiplies its compute budget by 12. Following this paper, roughly how should the model size and the token count change?

Chapter 7

Chinchilla: The Test Run

Follow the data through the 70 billion parameter model the authors trained to check their prediction, and see what they changed from Gopher

A prediction is cheap. A test is not. The authors have three fits that say Gopher is too big. To check, they must spend the same money again and train the model the fits recommend. This is the only point in the paper where two large models are compared at the same cost, and the whole claim rests on it.

Section 3 bracketed the best size for Gopher’s budget between 40 and 70 billion parameters. They chose the upper end, 70 billion, trained for 1.4 trillion tokens, “due to both dataset and computational efficiency considerations”. They named the model Chinchilla.

The same model, shrunk and re-aimed

Chinchilla uses the same architecture and setup as Gopher, a decoder-only transformer (a stack of blocks that read tokens left to right and predict the next one). Only a few things differ. Reading the list carefully matters, because each difference is a possible alternative explanation for the result.

  1. Data mix. Both train on MassiveText, but Chinchilla uses a slightly different weighting of its subsets, to suit the larger token count.
  2. Optimiser. Chinchilla uses AdamW, not Adam. The paper says this improves the loss and the downstream results after finetuning. A footnote adds that AdamW only passes Adam’s training performance about 80% of the way through the cosine cycle.
  3. Tokenizer. A slightly modified SentencePiece tokenizer that does not apply NFKC normalisation. 94.15% of tokens are the same as Gopher’s. The authors find it particularly helps maths and chemistry text.
  4. Precision. The forward and backward passes run in bfloat16, a compact number format. A float32 copy of the weights is kept in the optimiser state, so tiny updates are not lost.

Then come the size choices in the paper’s Table 4. Chinchilla has 80 layers, the same as Gopher. It has 64 attention heads instead of 128, a model width dmodel of 8,192 instead of 16,384, and a key and value size of 128 in both. The feed-forward layer is always 4 × dmodel wide. The maximum learning rate is 1 × 10−4, against 4 × 10−5 for Gopher, so the smaller model trains with a higher peak step size. The batch is 1.5 million tokens, doubled to 3 million midway through training. Gopher’s batch is 3 million, doubled to 6 million.

If the doubling falls at the halfway point (the paper says only “midway”), the number of optimiser steps is 0.7 trillion / 1.5 million + 0.7 trillion / 3 million = 466,667 + 233,333 = about 700,000 steps. That is our illustrative count, not a figure from the paper.

One batch through the model

Let B be the batch size in sequences and T the sequence length in tokens. Here is what happens to one batch, with the shape of the data written at each step. Step through it below.

  1. Token ids enter as a grid of integers, shape [B, T].
  2. The embedding table turns each id into a vector of 8,192 numbers, shape [B, T, 8192].
  3. In attention, three learned matrices of shape [8192, 8192] make queries, keys and values. The 8,192 numbers split into 64 heads of 128. Each head scores every token against each earlier one, shape [B, 64, T, T]. A fourth matrix mixes the heads back.
  4. In the feed-forward layer, a matrix of shape [8192, 32768] widens each vector four times, and one of shape [32768, 8192] narrows it again.
  5. Steps 3 and 4 repeat 80 times, each with its own weights, with the input added back after each (a residual connection).
  6. A final matrix turns each vector into one score per vocabulary word, shape [B, T, V], where V is the vocabulary size. A softmax (a step that turns the scores into probabilities that add to one) and the loss follow.

Step through the model

Pick a model, then press the stages in turn. The row shows where the data is. The text gives its shape and how many weights that stage holds.

Layer counts, widths, heads and key size are the paper’s Table 4. The weight counts use the usual count (attention 4d², feed-forward 8d², per block, ignoring biases and norms), which is ours. It reaches 64.4B for Chinchilla and 257.7B for Gopher: about 8% below the 70B and 280B the paper reports, and the paper does not itemise the remainder in this table.

Now the weight count. Attention holds four matrices of d × d, which is 4d². The feed-forward layer holds d × 4d and 4d × d, which is 8d². One block holds 12d². For Chinchilla d = 8,192, so one block holds 12 × 8,192² = 805 million weights, and 80 blocks hold 64.4 billion. The rest of the 70 billion is the part of the model outside the blocks that this table does not itemise.

pythondef block_params(d):
    attn = 4 * d * d              # Wq, Wk, Wv, Wo: four [d, d] matrices
    ffw  = 2 * d * (4 * d)        # [d, 4d] and [4d, d]
    return attn + ffw             # = 12 d^2

for name, L, d in [("Gopher", 80, 16384), ("Chinchilla", 80, 8192)]:
    print(name, L * block_params(d) / 1e9)   # 257.7, 64.4 (billions, blocks only)

# per generated token the forward pass costs about 2N FLOPs (Chapter 1):
# 2 * 70e9 = 140 GFLOPs, against 2 * 280e9 = 560 GFLOPs for Gopher

Why the smaller model is also cheaper afterwards

Training happens once. Using the model happens every day. Each token a model writes costs about 2N FLOPs, the forward pass of Chapter 1. At 70 billion parameters that is 140 billion FLOPs per token, a quarter of the 560 billion for Gopher. The weights also need a quarter of the memory: in 16-bit numbers, 140 gigabytes against 560. The paper says this makes Chinchilla easier to use on smaller hardware and cheaper to finetune. Training cost the same. Everything after is cheaper.

Chinchilla and Gopher cost about the same to train. Which of these differs most between them?

Chapter 8

Results, Ablations and Limits

Read what each table and figure shows, check which checks the authors ran, and mark where the claim stops

Someone tells you a model four times smaller beats a giant one at the same cost. What would make you believe it? One benchmark could be luck, or leakage. You would want many tests, some that cannot be gamed, and a plain list of where it failed. This chapter reads the paper’s evidence in that spirit.

Language modelling: better everywhere, with a warning

Figure 5 compares Chinchilla and Gopher on the subsets of The Pile, a mix of web, books, code and papers. The measure is bits per byte, the number of bits the model needs to encode each byte of text, so lower is better. Chinchilla wins on every subset. Against Jurassic-1 it wins on all but two: dm_mathematics and ubuntu_irc. On Wikitext103, its perplexity (a related score for how surprised the model is by the text, lower is better) is 7.16 against 7.75 for Gopher.

The authors add a caution. Chinchilla reads four times more text, so test text may have slipped into its training data. Leakage means the model has seen the answer in training, so the score is too good. For this reason the authors place more weight on tasks where leakage matters less.

MMLU: the headline

MMLU is a set of exam-style questions across 57 subjects, from chemistry to law. The test is 5-shot, meaning the prompt shows five solved examples first. Table 6 reports the averages. A random guesser gets 25.0%. The average human rater gets 34.5%. GPT-3 gets 43.9%, Gopher 60.0%, and Chinchilla 67.6%. Average human experts get 89.8%.

Table 6 also lists forecasts made by 73 competitive forecasters: 57.1% for the best result by June 2022 and 63.4% for June 2023. Chinchilla beat the 2023 forecast in March 2022. The abstract states 67.5%, while Table 6 and the text give 67.6%; we use the table. Figure 6 breaks it down by subject. Chinchilla is better on 51 of 57 subjects, equal on 2 and worse on 4: college_mathematics, econometrics, moral_scenarios and formal_logic. It reaches over 90% on four subjects, a first for any model according to the authors.

Other benchmarks

On BIG-bench, a collection of 62 hard and unusual tasks, Chinchilla averages 65.1% against 54.4% for Gopher, a gain of 10.7 points, and loses on only four tasks. On LAMBADA (predict the last word of a passage) it gets 77.4% against 74.5% for Gopher and 76.6% for MT-NLG 530B. On the reading tests RACE-h and RACE-m it gains over 10 points.

On closed-book question answering, the model must answer without looking anything up. Chinchilla reaches 31.5% on Natural Questions with 5 examples and 35.5% with 64, a new best for that setting. On TruthfulQA, which tests whether a model repeats common falsehoods, it gets 43.6% with no examples, against 29.5% for Gopher. The authors read this large gain, 14.1 points, as a sign that modelling the pre-training data better is enough to help. Pick a test below.

Compare the scores

Choose a benchmark. Bars show the score from the paper’s tables. Chinchilla has 70B parameters, Gopher 280B.

Numbers are from the paper: MMLU 5-shot (Table 6), BIG-bench average (Section 4.2.4), LAMBADA zero-shot (Table 7), Natural Questions dev 5-shot (Table 9), TriviaQA unfiltered test 5-shot (Table 9), TruthfulQA zero-shot (Section 4.2.5). The text gives a different Gopher figure for Natural Questions 5-shot (21%); we use the table. GPT-3 and MT-NLG appear only where the paper reports them.

Bias and toxicity

Chinchilla trains on the same data as Gopher with different weights, so the authors expect similar risks. On Winogender, a test of gender and occupation bias in pronoun resolution, Chinchilla resolves pronouns correctly more often than Gopher. They write that it “does suffer from bias and toxicity” but seems less affected than Gopher. They also call their evaluations not comprehensive.

The checks that held the method up

Several checks in the paper test the method, not the model. The cosine cycle length (Figure A1): overshooting the run by more than 25% degrades results, which is why every approach matches it. The learning-rate decay: decaying by 10 times or to zero differ little (10 times is slightly better), and decaying by only 5 times is clearly worse. The dataset: repeating the IsoFLOP analysis on C4 and on GitHub code finds the same equal scaling. And the agreement of three approaches with different errors, Chapters 3 to 5.

Where the claim stops

The authors list their own limits, and we add two of ours.

  1. Only two large runs. The cost means just Chinchilla and Gopher are compared at large scale, and there are no tests at sizes in between.
  2. The power-law assumption. The loss frontier shows some curvature (Appendix E). Fits to the early, middle and last thirds of the frontier disagree. The authors say this suggests they may still overestimate the best size for big budgets.
  3. One epoch only. All runs read each token at most once. Repeating data is left for future work.
  4. Data quality and scale. The authors guess, without testing it, that bigger datasets help only when they are good. More web text also brings more toxic and private text.
  5. Ours: the constants are touchy. With the paper’s printed, rounded constants, Chapter 5’s closed form gives 32 billion where the paper’s figure shows 40 billion.
  6. Ours: more than one thing changed. Chinchilla also changed the optimiser, tokenizer and data weights (Chapter 7), so the size-and-data effect is the prediction being tested, not the only difference.
Why do the authors place more weight on MMLU and BIG-bench than on language-modelling loss when comparing Chinchilla to Gopher?

Chapter 9

Field Guide

Keep the numbers, the terms, a checklist and the core code, and size a run of your own

The paper in one paragraph. Big language models of 2021 were trained on about 300 billion tokens whatever their size. Hoffmann and colleagues trained over 400 models and fitted the best split of a compute budget three ways: the envelope of training curves, IsoFLOP valleys, and a parametric loss. All three say that model size and tokens should grow in equal proportion, about 20 tokens for each parameter. They tested it by training Chinchilla, 70 billion parameters on 1.4 trillion tokens, at Gopher’s cost. It beat Gopher, GPT-3, Jurassic-1 and MT-NLG on most tasks and costs a quarter as much to run.

Size your own run

Drag the budget. The device gives the model size and the tokens, once by the rule of thumb and once by the fitted formula, and shows what the old 300-billion-token habit would have built instead.

Size a run

Pick a rule, then drag the budget. Warm is the model size, blue the tokens. The grey outline is what 300B tokens would buy at the same budget.

5.8 × 10²³

The first rule is N = √(C / 120), D = 20N, our reading of Table 3. The second is Equation 4 with the printed constants (Chapter 5), which gives smaller models at large budgets, as the paper notes for Approach 3. The default budget is Gopher’s, 5.76 × 1023 FLOPs, where the paper’s own answer is 67 to 70 billion parameters.

The numbers

QuantityValueWhere
Training computeC ≈ 6NDSection 3.3, Appendix F
Gopher budget5.76 × 1023 FLOPsFigure 2, Table 3
Gopher280B params, 300B tokensTable 1
Chinchilla70B params, 1.4T tokensTable 1, Section 4
Models trainedover 400, 70M to 16B, 5B to 500B tokensAbstract, Section 1
Exponents a, b (three approaches)0.50/0.50, 0.49/0.51, 0.46/0.54Table 2
Kaplan exponents0.73 / 0.27Table 2
Loss fit E, A, B1.69, 406.4, 410.7Appendix D.2, Eq. 10
Loss fit α, β0.34, 0.28Appendix D.2, Eq. 10
Tokens per parameterabout 20 (20.0 to 22.4 in Table 3)Our division of Table 3
MMLU, 5-shot67.6% against 60.0% for GopherTable 6
BIG-bench average65.1% against 54.4%Section 4.2.4
Wikitext103 perplexity7.16 against 7.75 (lower is better)Section 4.2.1

Terms in one line each

A checklist before you size a model

  1. Write down the budget C in FLOPs, not in days. Use C ≈ 6ND to move between them.
  2. Set the learning-rate schedule length to the planned token count. Never reuse a longer schedule.
  3. Start from about 20 tokens per parameter, then check it against a small IsoFLOP sweep of your own.
  4. Fit at the scale you care about. Small runs bend the line (Appendix E).
  5. Ask whether you have that many good tokens, and whether test sets leaked into them.
  6. If the model will be used for years, weigh the cost of running it, which falls with a smaller N.
  7. Report what else changed between runs: optimiser, tokenizer, data mix.

The core as code

pythonimport numpy as np

E, A, B, alpha, beta = 1.69, 406.4, 410.7, 0.34, 0.28      # Eq. 10, Appendix D.2

def loss(N, D):                                       # predicted final loss
    return E + A / N**alpha + B / D**beta

def optimal_split(C):                                 # Eq. 4: minimise loss s.t. 6*N*D = C
    G = (alpha * A / (beta * B)) ** (1 / (alpha + beta))
    a, b = beta / (alpha + beta), alpha / (alpha + beta)
    N = G * (C / 6) ** a
    return N, C / (6 * N)

def rule_of_thumb(C, r=20):                         # our reading of Table 3
    N = np.sqrt(C / (6 * r))
    return N, r * N

def isoflop_bottom(sizes, losses):                    # Approach 2: parabola in log10(N)
    p, q, r0 = np.polyfit(np.log10(sizes), losses, 2)
    return 10 ** (-q / (2 * p))

print(rule_of_thumb(5.76e23))     # (6.9e10, 1.39e12): Chinchilla's size
print(loss(70e9, 1.4e12), loss(280e9, 300e9))   # 1.937 vs 1.993

Where it sits in the field

This paper is the origin of the habit of training small models for very long. It is also the reason later reports quote tokens per parameter. The lecture on large multimodal models builds on it: Large Multimodal Models (built in parallel today). To go deeper on scaling laws, the CS336 lessons Scaling Laws Basics and Scaling Laws Details work through the fits. For the FLOP counting of Chapter 1, see PyTorch Resource Accounting. For how pretraining fits the whole pipeline, see CS224N Pretraining.

Credit. Training Compute-Optimal Large Language Models, Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals and Laurent Sifre, DeepMind, 2022. arXiv 2203.15556. Every explanation here is our own; every number is the paper’s unless marked ours or illustrative.

The takeaway. For a fixed budget, make the model smaller and the reading longer than the habit of 2021 said. About 20 tokens per parameter, and both grow as the square root of compute.

Now teach it back: open Teach and explain, on a whiteboard, why the best size is a valley. Or go back to the hero and find Chinchilla’s split by yourself.

You have a budget of 1024 FLOPs. Using about 20 tokens per parameter and C = 6ND, roughly how big a model do you train?
Chinchilla: Training Compute-Optimal Large Language Models
Read the paper · Back to the lecture: Large Multimodal Models · Back to Gleams