Scaling laws

Plain English. The empirical finding that model capability improves smoothly and predictably as you add compute, data and parameters — so predictably that labs can forecast a model's quality before spending the money to train it. Kaplan and colleagues published the original curves in 2020; the Chinchilla paper corrected the recipe, showing most models were under-fed on data relative to their size.

Why it moves money. Scaling laws are the underwriting document for the entire AI capex cycle. Hundreds of billions in data centres, chips and power make sense only if more compute reliably buys more capability. Every revision matters: Chinchilla shifted value towards data; efficiency recipes shift it away from raw scale; and any genuine flattening of the curve would strand a great deal of capital.

What to watch. Whether each new frontier model justifies its compute multiple over the last one, and whether the gains are coming from scale or from recipe — the second is much harder to defend.

From the signals. OpenAI's chief scientist said no lab can responsibly keep scaling flat out. A small-scale study found data drove 12x of pretraining gains against 3.7x for model tweaks. Magic claimed a ~50x more compute-efficient pretraining recipe.

Further reading. Kaplan et al., Scaling Laws for Neural Language Models (2020) · Hoffmann et al., the Chinchilla paper (2022)