A Review of Current Trends, Techniques, and Challenges in Large Language Models (LLMs)
Обзор современных тенденций, методов и проблем в области больших языковых моделей (LLM)
2024-03-01
SCID: 54.1/kqhr6mqj
Discuss with AI
in-context learninglarge language modelsnatural language generationnatural language understandingpre-training objectives
Figures from the paper
Abstract (AI)
Natural language Processing (NLP) has significantly transformed in the last decade, especially in the field of Language Modeling. Large Language Models (LLMs) have achieved SOTA performances on Natural Language Understanding (NLU) and Natural Language Generation (NLG) tasks by learning language representation in self-supervised ways. This paper provides a comprehensive survey to capture the progression of advances in Language Models. In this paper, we examine the different aspects of Language Models, which started with a few million parameters but have reached the size of a trillion in a very short time. We also look at how these LLMs transitioned from task-specific to task-independent to task-and-language-independent architectures. This paper extensively discusses different pre-training objectives, benchmarks, and transfer learning methods used in LLMs. It also examines different fine-tuning and In-Context learning techniques used in downstream tasks. It also explores how LLMs can perform well across many domains and datasets if sufficiently trained on a large and diverse dataset. Next, it discusses how, over time, the availability of cheap computational power and large datasets have improved LLM’s capabilities and raised new challenges. As part of our study, we also inspect LLMs from the lens of scalability to see how their performance is affected by the model’s depth, width, and data size. Lastly, we provide an empirical comparison of existing trends and techniques and a comprehensive analysis of where the field of LLM currently stand.
Key Findings
1
LLM architectures have evolved from task-specific systems to task-independent and eventually task-and-language-independent models.
2
Language models have rapidly scaled from a few million parameters to approximately trillion-parameter sizes, substantially reshaping their capabilities and architectures.
3
Large language models have achieved state-of-the-art performance on natural language understanding and generation through self-supervised language representation learning.
4
Model depth, width, and training-data size are identified as key scalability factors affecting LLM performance.
5
Sufficient training on large and diverse datasets enables LLMs to perform effectively across multiple domains and datasets, while increasing scale and computational availability introduce new challenges.
6
The survey analyzes pre-training objectives, benchmarks, transfer learning, fine-tuning, and in-context learning techniques used for downstream applications.
Research Object
Large language models (LLMs)
Research Subject
the progression, architectures, training and adaptation techniques, capabilities, scalability, performance, and challenges of LLMs
Publication Details
Publication Date
2024-03-01
Journal
Publisher
ISSN
Cited by
296
Open access PDF
Access Type
Author Information
Download PDF
Subscribe to digest
References available in scid.ai10
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer2019
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation2014
Distributed Representations of Words and Phrases and their Compositionality2013
LLaMA: Open and Efficient Foundation Language Models2023
Sequence to Sequence Learning with Neural Networks2014
Enriching Word Vectors with Subword Information2017
Training language models to follow instructions with human feedback2022
GLM: General Language Model Pretraining with Autoregressive Blank Infilling2022
GPT-NeoX-20B: An Open-Source Autoregressive Language Model2022
Evaluating the Effectiveness of Large Language Models in Representing Textual Descriptions of Geometry and Spatial Relations (Short Paper)2023