OpenAI wrote a paper on what happens to model performance as you scale parameters, data, and compute. It's called Scaling Laws for Neural Language Models, 2020.
scaling laws
Scaling laws are simple predictive rules for how a language model's loss behaves as you grow it. The practical payoff: tune on small, cheap models, then extrapolate to the large one and nail it in a single run instead of guessing hyperparameters at full scale.
Kaplan et al. (2020) quantified this with three power laws:
$$L(N) \sim N^{-0.076}$$
$$L(D) \sim D^{-0.095}$$
$$L(C) \sim C^{-0.050}$$
where N = number of parameters, D = dataset size (tokens), C = compute budget (FLOPs), and L = cross-entropy loss. Loss drops as a power law as you add more parameters, data, or compute, but the small exponents mean diminishing returns: each 10x increase in any factor buys only a modest drop in loss.
Two things followed from this. GPT-3 turned out to be undertrained, and Chinchilla (40% of the parameters of GPT-3, 4x the data) beat it by spending the same compute budget more wisely.
data
Loss versus dataset size is also a power law, linear on a log-log plot.
Data composition shifts the offset, not the slope, so you can run data-selection experiments on a small model and trust they transfer to the big one.
Repeating data helps up to about 4 epochs, then returns fall off fast. That sets up the real tradeoff: repeat high-quality data, or pull in new lower-quality data.
how to design a huge LM
- Architecture: Transformer over LSTM; loss keeps dropping as you add parameters. MoE is the only thing that consistently beats a vanilla transformer.
- Optimizer: Adam over SGD once you train for many epochs (adaptive step sizes instead of one fixed rate).
- Depth: 1 vs 2 layers is night and day; past ~6 layers it plateaus.
- Batch size: bigger batches mean more gradient signal per step, up to a critical batch size; past it you spend compute for little gain.
recap
You're always budget-constrained, so you pick which of the three knobs to turn: parameters, data, or compute. You can turn all three - there's no impossibility theorem - but nobody has infinite money. The Chinchilla paper showed most labs were turning the wrong knob. They were making models too big and not training them on enough data. Chinchilla (70B params, 4x the data of GPT-3) outperformed the much larger GPT-3 by allocating the compute budget more wisely.
Loss ~ f(parameters, data, compute) - all three matter, but how you balance them matters more.
Different labs make different bets on this tradeoff. ChatGPT scales up parameters and broad data with massive compute - optimizing for generality. DeepSeek goes the other way: fewer parameters, higher-quality data, and more inference-time compute - optimizing for reasoning efficiency. Both obey the same scaling laws, they just allocate their budgets differently.
limitations and what's next
The catch: we're running out of high-quality text. Two escape routes are in play. Synthetic data (AlphaGo learned entirely from self-play) and reasoning models like o1 that spend more compute at inference, thinking longer to answer better, which is its own new scaling axis. New architectures are the other hope, but nothing has unseated the transformer yet, usually because the alternatives aren't numerically stable.
