ERNIE 2.0: A Continual Pre-Training Framework for Language Understanding
ERNIE 2.0: Непрерывная рамочная схема дополнительного предобучения для понимания языка
2020-04-03
SCID: 54.1/2meu9vem
Discuss with AI
ERNIE 2.0GLUE benchmarkcontinual multi-task learningcontinual pre-traininglexical syntactic semantic pre-training tasks
Figures from the paper
Abstract (AI)
Recently pre-trained models have achieved state-of-the-art results in various language understanding tasks. Current pre-training procedures usually focus on training the model with several simple tasks to grasp the co-occurrence of words or sentences. However, besides co-occurring information, there exists other valuable lexical, syntactic and semantic information in training corpora, such as named entities, semantic closeness and discourse relations. In order to extract the lexical, syntactic and semantic information from training corpora, we propose a continual pre-training framework named ERNIE 2.0 which incrementally builds pre-training tasks and then learn pre-trained models on these constructed tasks via continual multi-task learning. Based on this framework, we construct several tasks and train the ERNIE 2.0 model to capture lexical, syntactic and semantic aspects of information in the training data. Experimental results demonstrate that ERNIE 2.0 model outperforms BERT and XLNet on 16 tasks including English tasks on GLUE benchmarks and several similar tasks in Chinese. The source codes and pre-trained models have been released at https://github.com/PaddlePaddle/ERNIE.
Key Findings
1
ERNIE 2.0 is a continual pre-training framework that incrementally builds pre-training tasks and uses continual multi-task learning.
2
ERNIE 2.0 trained on these tasks outperforms BERT and XLNet on 16 tasks, including English GLUE benchmarks and several Chinese tasks.
3
The authors provide source code and pre-trained models publicly at https://github.com/PaddlePaddle/ERNIE.
4
The framework constructs tasks designed to capture lexical, syntactic, and semantic information beyond simple co-occurrence signals.
Research Object
ERNIE 2.0 continual pre-training framework and its pre-trained language model
Research Subject
Learning and capturing lexical, syntactic and semantic information via incremental pre-training tasks and continual multi-task learning to improve language understanding performance
Publication Details
Publication Date
2020-04-03
Journal
Publisher
ISSN
Cited by
766
Open access PDF
Access Type
Author Information
Download PDF
Subscribe to digest
References available in scid.ai5
Exploiting Generative AI to Scale up Intelligent Tutoring Systems2023
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank2013
Transformer-XL: Attentive Language Models beyond a Fixed-Length Context2019
Continual lifelong learning with neural networks: A review2019
Qualitative Spatial Reasoning over Questions (Short Paper)2022
Cited by4
A Survey on Knowledge Graphs: Representation, Acquisition, and Applications2021
RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering2021
Lexicon Enhanced Chinese Sequence Labeling Using BERT Adapter2021
RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering2020