Scaling Laws for Neural Language Models
Законы масштабирования нейронных языковых моделей
2020-01-23
SCID: 54.1/yk582ph2
Discuss with AI
agentic AI interventionsexecutable lower certificatesfrontier-scale deploymentneural language modelstransport-validity theory
Figures from the paper
Abstract (AI)
We study empirical scaling laws for language model performance on the cross-entropy loss. The loss scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude. Other architectural details such as network width or depth have minimal effects within a wide range. Simple equations govern the dependence of overfitting on model/dataset size and the dependence of training speed on model size. These relationships allow us to determine the optimal allocation of a fixed compute budget. Larger models are significantly more sample-efficient, such that optimally compute-efficient training involves training very large models on a relatively modest amount of data and stopping significantly before convergence.
Key Findings
1
It introduces executable lower certificates, including replayable identified sets, descriptor-growth audits, witness-cover transport certificates, branch-local evaluation bridges, and confidence-calibrated audit rules.
2
It proves that naive extrapolation can fail through sign reversal under bottleneck-weight shifts and through vacuous observable closeness when descriptors omit sign-relevant coordinates.
3
The framework targets operationally isolatable interventions with event-local logged channels, bounded spillover, and replayable extraction maps.
4
The paper develops a transport-validity theory for deciding when comparative gains from small-scale agentic AI interventions can support frontier-scale trials.
5
The resulting finite, machine-readable framework supports falsifiable scale-up decisions and frontier-budget allocation across interventions while retaining the need for frontier experimentation.
Research Object
operationally isolatable agentic AI interventions evaluated from small-scale systems for frontier-scale deployment
Research Subject
transport validity of comparative gains under scaling, including failure modes, executable lower certificates, and frontier-budget allocation
Publication Details
Publication Date
2020-01-23
Journal
Publisher
ISSN
Cited by
1753
Open access PDF
Access Type
Author Information
Download PDF
Subscribe to digest
References available in scid.ai2
Cited by20
LLaMA: Open and Efficient Foundation Language Models2023
A Survey of Large Language Models2026
Retrieval-Augmented Generation for Large Language Models: A Survey2023
Large language models encode clinical knowledge2023
Scalable Diffusion Models with Transformers2023
A Comprehensive Overview of Large Language Models2025
Swin Transformer V2: Scaling Up Capacity and Resolution2022
Generative AI at Work2024
BloombergGPT: A Large Language Model for Finance2023
Recent Advances in Natural Language Processing via Large Pre-trained Language Models: A Survey2023
Autonomous chemical research with large language models2023
Scaling Vision Transformers2022
Atlas: Few-shot Learning with Retrieval Augmented Language Models2022
Retrieval-Augmented Generation for AI-Generated Content: A Survey2026
Can Large Language Models Transform Computational Social Science?2023
ChatDev: Communicative Agents for Software Development2024
Large Language Models for Software Engineering: Survey and Open Problems2023
Large Language Models for Cyber Security: A Systematic Literature Review2025
Generative artificial intelligence in supply chain and operations management: a capability-based framework for analysis and implementation2024
GPT-NeoX-20B: An Open-Source Autoregressive Language Model2022