Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Исследование пределов трансферного обучения с помощью унифицированного преобразователя текста в текст
2019-10-23
SCID: 54.1/y4yj9rjq
Discuss with AI
Colossal Clean Crawled Corpuslanguage understanding tasksnatural language processingtext-to-text frameworktransfer learning
Figures from the paper
Abstract (AI)
Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts all text-based language problems into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new ``Colossal Clean Crawled Corpus'', we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our data set, pre-trained models, and code.
Key Findings
1
A systematic study compares pre-training objectives, model architectures, unlabeled datasets, transfer strategies, and related factors across dozens of language-understanding tasks.
2
Combining the study’s findings with increased model scale and the Colossal Clean Crawled Corpus achieves state-of-the-art results on benchmarks in summarization, question answering, text classification, and other areas.
3
The authors release the Colossal Clean Crawled Corpus, pretrained models, and code to support future NLP transfer-learning research.
4
The paper introduces a unified text-to-text framework that represents diverse NLP problems using a single task format.
Research Object
Unified text-to-text transformer framework for natural language processing (including the Colossal Clean Crawled Corpus and pre-trained models)
Research Subject
the effectiveness and limits of pre-training objectives, architectures, unlabeled datasets, and transfer approaches across diverse NLP tasks
Publication Details
Publication Date
2019-10-23
Journal
Publisher
ISSN
Cited by
31232
Open access PDF
Access Type
Author Information
Download PDF
Subscribe to digest
References available in scid.ai9
Deep Residual Learning for Image Recognition2016
ImageNet: A large-scale hierarchical image database2009
Distilling the Knowledge in a Neural Network2015
Distributed Representations of Words and Phrases and their Compositionality2013
Sequence to Sequence Learning with Neural Networks2014
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank2013
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation2016
Self-Attention with Relative Position Representations2018
TinyBERT: Distilling BERT for Natural Language Understanding2020
Cited by20
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows2021
Transformers: State-of-the-Art Natural Language Processing2020
A Survey of Large Language Models2026
Dense Passage Retrieval for Open-Domain Question Answering2020
Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing2022
Large language models encode clinical knowledge2023
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks2020
Lost in the Middle: How Language Models Use Long Contexts2024
A Comprehensive Overview of Large Language Models2025
HuggingFace's Transformers: State-of-the-art Natural Language Processing2019
CodeBERT: A Pre-Trained Model for Programming and Natural Languages2020
Swin Transformer V2: Scaling Up Capacity and Resolution2022
Unifying Large Language Models and Knowledge Graphs: A Roadmap2024
ChatGPT: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope2023
BloombergGPT: A Large Language Model for Finance2023
Pre-Trained Image Processing Transformer2021
Video Swin Transformer2022
🧜Siren’s Song in the AI Ocean: A Survey on Hallucination in Large Language Models2025
WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing2022
TinyBERT: Distilling BERT for Natural Language Understanding2020