Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

Исследование пределов трансферного обучения с помощью унифицированного преобразователя текста в текст
Noam Shazeer, Katherine Lee, Sharan Narang, Colin Raffel, Adam Paul Roberts, Peter J. Liu, Michael S Matena, Yanqi Zhou, Wei Li, Peter J. Liu, Adam Roberts, Roberts, Adam
2019-10-23

Colossal Clean Crawled Corpuslanguage understanding tasksnatural language processingtext-to-text frameworktransfer learning
Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts all text-based language problems into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new ``Colossal Clean Crawled Corpus'', we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our data set, pre-trained models, and code.
1
A systematic study compares pre-training objectives, model architectures, unlabeled datasets, transfer strategies, and related factors across dozens of language-understanding tasks.
2
Combining the study’s findings with increased model scale and the Colossal Clean Crawled Corpus achieves state-of-the-art results on benchmarks in summarization, question answering, text classification, and other areas.
3
The authors release the Colossal Clean Crawled Corpus, pretrained models, and code to support future NLP transfer-learning research.
4
The paper introduces a unified text-to-text framework that represents diverse NLP problems using a single task format.

Unified text-to-text transformer framework for natural language processing (including the Colossal Clean Crawled Corpus and pre-trained models)

the effectiveness and limits of pre-training objectives, architectures, unlabeled datasets, and transfer approaches across diverse NLP tasks

Publication Details
Publication Date
2019-10-23
Journal
Publisher
ISSN
Cited by
31232
Access Type
Author Information
Authors
Noam Shazeer
Katherine Lee
Sharan Narang
Colin Raffel
Adam Paul Roberts
Peter J. Liu
Michael S Matena
Yanqi Zhou
Wei Li
Peter J. Liu
Adam Roberts
Roberts, Adam
Explore further
Open the scid.ai AI chat with a ready-made request: it will find papers on a similar topic and help build a literature review.
Find similar papers in the chat →
Make a presentation
100%