A Review of Current Trends, Techniques, and Challenges in Large Language Models (LLMs)

Обзор современных тенденций, методов и проблем в области больших языковых моделей (LLM)
Rajvardhan Patil, Venkat Naidu Gudivada
2024-03-01

in-context learninglarge language modelsnatural language generationnatural language understandingpre-training objectives
Natural language Processing (NLP) has significantly transformed in the last decade, especially in the field of Language Modeling. Large Language Models (LLMs) have achieved SOTA performances on Natural Language Understanding (NLU) and Natural Language Generation (NLG) tasks by learning language representation in self-supervised ways. This paper provides a comprehensive survey to capture the progression of advances in Language Models. In this paper, we examine the different aspects of Language Models, which started with a few million parameters but have reached the size of a trillion in a very short time. We also look at how these LLMs transitioned from task-specific to task-independent to task-and-language-independent architectures. This paper extensively discusses different pre-training objectives, benchmarks, and transfer learning methods used in LLMs. It also examines different fine-tuning and In-Context learning techniques used in downstream tasks. It also explores how LLMs can perform well across many domains and datasets if sufficiently trained on a large and diverse dataset. Next, it discusses how, over time, the availability of cheap computational power and large datasets have improved LLM’s capabilities and raised new challenges. As part of our study, we also inspect LLMs from the lens of scalability to see how their performance is affected by the model’s depth, width, and data size. Lastly, we provide an empirical comparison of existing trends and techniques and a comprehensive analysis of where the field of LLM currently stand.
1
LLM architectures have evolved from task-specific systems to task-independent and eventually task-and-language-independent models.
2
Language models have rapidly scaled from a few million parameters to approximately trillion-parameter sizes, substantially reshaping their capabilities and architectures.
3
Large language models have achieved state-of-the-art performance on natural language understanding and generation through self-supervised language representation learning.
4
Model depth, width, and training-data size are identified as key scalability factors affecting LLM performance.
5
Sufficient training on large and diverse datasets enables LLMs to perform effectively across multiple domains and datasets, while increasing scale and computational availability introduce new challenges.
6
The survey analyzes pre-training objectives, benchmarks, transfer learning, fine-tuning, and in-context learning techniques used for downstream applications.

Large language models (LLMs)

the progression, architectures, training and adaptation techniques, capabilities, scalability, performance, and challenges of LLMs

Publication Details
Publication Date
2024-03-01
Journal
Publisher
ISSN
Cited by
296
Access Type
Author Information
Authors
Rajvardhan Patil
Venkat Naidu Gudivada
Explore further
Open the scid.ai AI chat with a ready-made request: it will find papers on a similar topic and help build a literature review.
Find similar papers in the chat →
Make a presentation
100%