Parameter-efficient fine-tuning in large language models: a survey of methodologies

Параметроэффективная тонкая настройка больших языковых моделей: обзор методологий
Luping Wang, Sheng Chen, Linnan Jiang, Shu Pan, Runze Cai, Sen Yang, Fei Yang
2025-05-02

GPU memorydownstream task adaptationlarge language modelsparameter-efficient fine-tuningpre-trained models
Abstract The large models, as predicted by scaling law forecasts, have made groundbreaking progress in many fields, particularly in natural language generation tasks, where they have approached or even surpassed human levels. However, the unprecedented scale of their parameters brings significant computational and storage costs. These large models require substantial computational resources and GPU memory to operate. When adapting large models to specific downstream tasks, their massive parameter scale poses a significant challenge in fine-tuning on hardware platforms with limited computational power and GPU memory. To address this issue, Parameter-Efficient Fine-Tuning (PEFT) offers a practical solution by efficiently adjusting the parameters of large pre-trained models to suit various downstream tasks. Specifically, PEFT adjusts the parameters of pre-trained large models to adapt to specific tasks or domains, minimizing the introduction of additional parameters and the computational resources required. This review mainly introduces the preliminary knowledge of PEFT, the core ideas and principles of various PEFT algorithms, the applications of PEFT, and potential future research directions. By reading this review, we believe that interested parties can quickly grasp the PEFT methodology, thereby accelerating its development and innovation.
1
Large language models achieve groundbreaking performance in natural language generation, approaching or surpassing human levels in some tasks.
2
PEFT provides a practical approach for adapting large models on hardware with limited computational power and GPU memory.
3
Parameter-efficient fine-tuning adapts pretrained large models to specific tasks or domains while minimizing added parameters and computational requirements.
4
The survey reviews PEFT fundamentals, algorithmic principles, applications, and potential future research directions to support further methodological development.
5
The unprecedented parameter scale of large models creates substantial computational, GPU-memory, and storage costs, complicating downstream fine-tuning.

large pre-trained language models adapted to downstream tasks

parameter-efficient fine-tuning methodologies, including their principles, applications, and computational and storage efficiency

Publication Details
Publication Date
2025-05-02
Journal
Publisher
ISSN
Access Type
Author Information
Authors
Luping Wang
Sheng Chen
Linnan Jiang
Shu Pan
Runze Cai
Sen Yang
Fei Yang
Explore further
Open the scid.ai AI chat with a ready-made request: it will find papers on a similar topic and help build a literature review.
Find similar papers in the chat →
Make a presentation
100%