A Survey of Large Language Models
Обзор больших языковых моделей
2026-05-09
SCID: 54.1/c3gnfpzn
Discuss with AI
in-context learninglarge language modelsmodel evaluationpost-trainingpre-training
Figures from the paper
Abstract (AI)
Abstract The rapid evolution of large language models (LLMs) has driven a transformative shift in artificial intelligence (AI), reshaping both research paradigms and practical applications. Distinguished from their predecessors by unprecedented scale and advanced capabilities, LLMs necessitate new frameworks for understanding their development, behavior, and societal impact. This survey systematically reviews recent advancements in LLM techniques across four key dimensions: (1) pre-training methodologies, which establish core model capabilities through large-scale self-supervised training, architectural innovations, and data curation strategies; (2) post-training techniques, including supervised fine-tuning and reinforcement learning, which adapt foundational models to downstream tasks and enhance their alignment and safety; (3) utilization strategies, such as in-context learning, prompt engineering, and agentic reasoning, that optimize real-world deployment and enable effective interaction with external environments; and (4) evaluation methods, encompassing benchmarks for key ability dimensions such as core language capabilities, reasoning, and safety, which support comprehensive and reliable assessment of model performance. Additionally, we identify critical research issues, including those concerning theoretical foundations, efficient scaling, alignment, and agentic capability, and highlight the open challenges they present. By synthesizing state-of-the-art insights and emerging trends, this survey aims to provide a systematic and comprehensive framework for understanding the trajectory, current limitations, and future directions of LLM progress.
Key Findings
1
Post-training methods, including supervised fine-tuning and reinforcement learning, adapt foundation models while improving alignment and safety.
2
Pre-training advances rely on large-scale self-supervised learning, architectural innovations, and data curation to establish foundational model capabilities.
3
The survey identifies unresolved challenges involving theoretical foundations, efficient scaling, alignment, agentic capabilities, and reliable evaluation of LLMs.
4
The survey organizes large language model research into four dimensions: pre-training, post-training, utilization, and evaluation.
5
Utilization strategies such as in-context learning, prompt engineering, and agentic reasoning support effective deployment and interaction with external environments.
Research Object
Large language models (LLMs)
Research Subject
LLM development, behavior, capabilities, evaluation, alignment, safety, and societal impact across pre-training, post-training, utilization, and assessment
Publication Details
Publication Date
2026-05-09
Journal
Publisher
ISSN
Cited by
8718
Open access PDF
Access Type
Author Information
Download PDF
Subscribe to digest
References available in scid.ai20
Deep Residual Learning for Image Recognition2016
PyTorch: An Imperative Style, High-Performance Deep Learning Library2019
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer2019
Distributed Representations of Words and Phrases and their Compositionality2013
LLaMA: Open and Efficient Foundation Language Models2023
WordNet: a lexical database for English1995
Transformers: State-of-the-Art Natural Language Processing2020
Retrieval-Augmented Generation for Large Language Models: A Survey2023
Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing2022
Recurrent neural network based language model2010
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation2016
Large language models encode clinical knowledge2023
Training language models to follow instructions with human feedback2022
Lost in the Middle: How Language Models Use Long Contexts2024
Transformer-XL: Attentive Language Models beyond a Fixed-Length Context2019
A survey on large language model based autonomous agents2024
Generative Agents: Interactive Simulacra of Human Behavior2023
PaLM: Scaling Language Modeling with Pathways2022
Self-Attention with Relative Position Representations2018
Convolutional 2D Knowledge Graph Embeddings2018
Cited by20
A Survey on Evaluation of Large Language Models2024
A survey on large language model based autonomous agents2024
A Comprehensive Overview of Large Language Models2025
Unifying Large Language Models and Knowledge Graphs: A Roadmap2024
🧜Siren’s Song in the AI Ocean: A Survey on Hallucination in Large Language Models2025
A survey on large language model (LLM) security and privacy: The Good, The Bad, and The Ugly2024
Recommender Systems in the Era of Large Language Models (LLMs)2024
Active Retrieval Augmented Generation2023
Large Language Models for Software Engineering: A Systematic Literature Review2024
A Review on Large Language Models: Architectures, Applications, Taxonomies, Open Issues and Challenges2024
A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models2024
MM-LLMs: Recent Advances in MultiModal Large Language Models2024
A Survey on Large Language Models: Applications, Challenges, Limitations, and Practical Usage2023
Software Testing With Large Language Models: Survey, Landscape, and Vision2024
A Survey on Graph Neural Networks for Time Series: Forecasting, Classification, Imputation, and Anomaly Detection2024
Large Language Models for Cyber Security: A Systematic Literature Review2025
Exploring Large Language Model‐Based Intelligent Agents: Definitions, Methods, and Prospects2026
Large language models in medical and healthcare fields: applications, advances, and challenges2024
A comprehensive survey of ChatGPT: Advancements, applications, prospects, and challenges2023
A Survey on Large Language Models for Code Generation2025