Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing
Предварительное обучение, промптинг и предсказание: систематический обзор методов промптинга в обработке естественного языка
2022-09-14
SCID: 54.1/9xc62kpu
Discuss with AI
few-shot learningnatural language processingpre-trained language modelsprompt-based learningzero-shot learning
Figures from the paper
Abstract (AI)
This article surveys and organizes research works in a new paradigm in natural language processing, which we dub “prompt-based learning.” Unlike traditional supervised learning, which trains a model to take in an input x and predict an output y as P ( y|x ), prompt-based learning is based on language models that model the probability of text directly. To use these models to perform prediction tasks, the original input x is modified using a template into a textual string prompt x′ that has some unfilled slots, and then the language model is used to probabilistically fill the unfilled information to obtain a final string x̂ , from which the final output y can be derived. This framework is powerful and attractive for a number of reasons: It allows the language model to be pre-trained on massive amounts of raw text, and by defining a new prompting function the model is able to perform few-shot or even zero-shot learning, adapting to new scenarios with few or no labeled data. In this article, we introduce the basics of this promising paradigm, describe a unified set of mathematical notations that can cover a wide variety of existing work, and organize existing work along several dimensions, e.g., the choice of pre-trained language models, prompts, and tuning strategies. To make the field more accessible to interested beginners, we not only make a systematic review of existing works and a highly structured typology of prompt-based concepts but also release other resources, e.g., a website NLPedia–Pretrain including constantly updated survey and paperlist.
Key Findings
1
Prompt-based learning differs from traditional supervised prediction by leveraging language models that directly model text probabilities rather than P(y|x).
2
Prompting enables few-shot and zero-shot adaptation to new scenarios by exploiting large-scale pretraining and task-specific prompting functions.
3
The authors contribute a systematic review, structured typology, and continuously updated NLPedia–Pretrain website and paper list.
4
The paper defines prompt-based learning as modifying inputs into textual templates with unfilled slots, which language models complete to derive task outputs.
5
The survey provides unified mathematical notation and organizes prompt-based research by pretrained models, prompt designs, and tuning strategies.
Research Object
Prompt-based learning paradigm in natural language processing (using pre-trained language models and textual prompts to perform prediction tasks)
Research Subject
the methods, formulations, and strategies for designing, tuning, and applying prompts to enable few-shot and zero-shot prediction
Publication Details
Publication Date
2022-09-14
Journal
Publisher
ISSN
Cited by
5724
Open access PDF
Access Type
Author Information
Download PDF
Subscribe to digest
References available in scid.ai7
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer2019
Representation Learning: A Review and New Perspectives2013
Neural Machine Translation by Jointly Learning to Align and Translate2014
Convolutional Neural Networks for Sentence Classification2014
Speech recognition with deep recurrent neural networks2013
Evaluating Large Language Models Trained on Code2021
REALM: Retrieval-Augmented Language Model Pre-Training2020
Cited by20
A Survey of Large Language Models2026
Large language models encode clinical knowledge2023
Embracing the future of Artificial Intelligence in the classroom: the relevance of AI literacy, prompt engineering, and critical thinking in modern education2024
Generative AI2023
BioGPT: generative pre-trained transformer for biomedical text generation and mining2022
Recent Advances in Natural Language Processing via Large Pre-trained Language Models: A Survey2023
Autonomous chemical research with large language models2023
Multimodal Learning With Transformers: A Survey2023
Unleashing the potential of prompt engineering for large language models2025
Design Guidelines for Prompt Engineering Text-to-Image Generative Models2022
Active Retrieval Augmented Generation2023
A Review on Large Language Models: Architectures, Applications, Taxonomies, Open Issues and Challenges2024
Self-Supervised Speech Representation Learning: A Review2022
ExpeL: LLM Agents Are Experiential Learners2024
Regulating ChatGPT and other Large Generative AI Models2023
Machine Learning Methods for Small Data Challenges in Molecular Science2023
Large Language Models for Cyber Security: A Systematic Literature Review2025
AI literacy and its implications for prompt engineering strategies2024
A Survey on Large Language Models for Code Generation2025
The Biases of Pre-Trained Language Models: An Empirical Study on Prompt-Based Sentiment Analysis and Emotion Detection2022