Scaling Laws for Neural Language Models

Законы масштабирования нейронных языковых моделей
Alec Radford, Tom Henighan, Dario Amodei, Sam McCandlish, Jared Kaplan A, Jeffrey Wu, T. B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Tom Henighan, Brown, Tom B.
2020-01-23

agentic AI interventionsexecutable lower certificatesfrontier-scale deploymentneural language modelstransport-validity theory
We study empirical scaling laws for language model performance on the cross-entropy loss. The loss scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude. Other architectural details such as network width or depth have minimal effects within a wide range. Simple equations govern the dependence of overfitting on model/dataset size and the dependence of training speed on model size. These relationships allow us to determine the optimal allocation of a fixed compute budget. Larger models are significantly more sample-efficient, such that optimally compute-efficient training involves training very large models on a relatively modest amount of data and stopping significantly before convergence.
1
It introduces executable lower certificates, including replayable identified sets, descriptor-growth audits, witness-cover transport certificates, branch-local evaluation bridges, and confidence-calibrated audit rules.
2
It proves that naive extrapolation can fail through sign reversal under bottleneck-weight shifts and through vacuous observable closeness when descriptors omit sign-relevant coordinates.
3
The framework targets operationally isolatable interventions with event-local logged channels, bounded spillover, and replayable extraction maps.
4
The paper develops a transport-validity theory for deciding when comparative gains from small-scale agentic AI interventions can support frontier-scale trials.
5
The resulting finite, machine-readable framework supports falsifiable scale-up decisions and frontier-budget allocation across interventions while retaining the need for frontier experimentation.

operationally isolatable agentic AI interventions evaluated from small-scale systems for frontier-scale deployment

transport validity of comparative gains under scaling, including failure modes, executable lower certificates, and frontier-budget allocation

Publication Details
Publication Date
2020-01-23
Journal
Publisher
ISSN
Cited by
1753
Access Type
Author Information
Authors
Alec Radford
Tom Henighan
Dario Amodei
Sam McCandlish
Jared Kaplan A
Jeffrey Wu
T. B. Brown
Benjamin Chess
Rewon Child
Scott Gray
Tom Henighan
Brown, Tom B.
Explore further
Open the scid.ai AI chat with a ready-made request: it will find papers on a similar topic and help build a literature review.
Find similar papers in the chat →
Make a presentation
100%