Why bigger worked: scaling laws

The math, lightlyCopy Anchor Link

Test loss L as a function of model size N follows roughly L(N) = (Nc / N)^α, with a small exponent α, around 0.07 for the early OpenAI fits. A power law is a straight line on a log-log plot. Each time you multiply N by 10, loss drops by the same factor, 10^(-α), which is about 15 percent for α = 0.07.

Chinchilla refined this: for a compute budget C, the best split gives parameters and tokens that grow together, and a good rule is about 20 training tokens per parameter. A 70 billion parameter model should see around 1.4 trillion tokens.

ExampleCopy Anchor Link

Suppose a 1B model has loss 3.0. With α = 0.07, a 10B model has loss 3.0 × 10^(-0.07) = 3.0 × 0.851 = 2.55. Another 10x gives 2.17. Each 10x in model size, with data growing alongside, costs about 100 times more compute for a smaller absolute gain, which is why labs also look for gains elsewhere: better data, reasoning at inference time and tools.

What to take from itCopy Anchor Link

Scaling is a trend, not a guarantee of every skill. It tells you that bigger runs reliably improve average loss, not when a particular ability will appear. It also explains why training tokens are as important as parameters.

Reading the exponentCopy Anchor Link

An exponent of 0.07 sounds tiny, and it is. It means progress is steady but slow in absolute terms. To halve the loss with that exponent you would need to multiply the model size by about 2^(1/0.07), which is around 20,000 times. That is why labs build enormous data centers and why they also hunt for other gains, like better training data and letting models think longer at answer time.

A worked example with ChinchillaCopy Anchor Link

You have a budget to train a model. Option A: 280 billion parameters on 300 billion tokens. Option B: 70 billion parameters on 1.4 trillion tokens. Chinchilla found B wins at the same cost, because A was too big for how little data it saw. Rule of thumb: about 20 tokens of training text per parameter.

Planning a run with the 20-to-1 ruleCopy Anchor Link

Invented budget for illustration: you want a model with 10 billion parameters. The rule of thumb says about 20 tokens per parameter, so you should plan for 200 billion training tokens (10 billion × 20).

Now double the model to 20 billion parameters. The data should double too, to 400 billion tokens. Training compute is roughly proportional to parameters times tokens, so it rises by 2 × 2 = 4 times. Growing both together makes costs climb faster than either number alone.

Labs often train smaller models on more tokens than this rule suggests. The reasoning is that a model is trained once but answered from millions of times, so a smaller model is cheaper to serve. The rule is a guide for the cheapest route to a given loss, not a law about what to build.

When scaling stops payingCopy Anchor Link

The formula keeps falling, but real limits exist. Text is partly unpredictable, so loss cannot reach zero. Good text is finite, and repeating the same data gives less benefit than fresh data. Each 10x step costs about 100 times more compute (more parameters and more data) for a smaller gain, as the example above showed. This is why scaling is one lever among several.

What goes wrong when you read a scaling claimCopy Anchor Link

The symptom is a confident line on a chart extended far past what was measured. Two checks help.

1. Look at the y-axis. Does it measure average loss, or a specific skill such as arithmetic? The curve only promises the first.
2. Look at the range. A fit over a few orders of magnitude may not hold a hundred times further out.

A straight line on a log-log chart is a fit to past runs, not a guarantee.

Try it yourselfCopy Anchor Link

With α = 0.07 and a 1 billion parameter model at loss 3.0, compute the loss for a model 1,000 times bigger. That is 3.0 × 10^(-0.21), and 10^(-0.21) is about 0.617, so the loss is about 1.85. Expect a drop of roughly 38 percent for a thousandfold increase. Compare that with the cost and decide whether the gain feels large.

Common mistakeCopy Anchor Link

Treating scaling laws as a promise that any skill will appear at some size. They predict average error, not when a specific ability such as reliable arithmetic will show up.