Transformer-XL: Attentive Language Models beyond a Fixed-Length Context
Transformer-XL: внимательные языковые модели за пределами контекста фиксированной длины
2019-01-01
SCID: 54.1/4hsyvnq2
Discuss with AI
Transformer-XLcontext fragmentationlong-range dependencynovel positional encodingsegment-level recurrence
Figures from the paper
Abstract (AI)
Transformers have a potential of learning longer-term dependency, but are limited by a fixed-length context in the setting of language modeling.We propose a novel neural architecture Transformer-XL that enables learning dependency beyond a fixed length without disrupting temporal coherence.It consists of a segment-level recurrence mechanism and a novel positional encoding scheme.Our method not only enables capturing longer-term dependency, but also resolves the context fragmentation problem.As a result, Transformer-XL learns dependency that is 80% longer than RNNs and 450% longer than vanilla Transformers, achieves better performance on both short and long sequences, and is up to 1,800+ times faster than vanilla Transformers during evaluation.Notably, we improve the state-ofthe-art results of bpc/perplexity to 0.99 on en-wiki8, 1.08 on text8, 18.3 on WikiText-103, 21.8 on One Billion Word, and 54.5 on Penn Treebank (without finetuning).When trained only on WikiText-103, Transformer-XL manages to generate reasonably coherent, novel text articles with thousands of tokens.Our code, pretrained models, and hyperparameters are available in both Tensorflow and PyTorch 1 .
Key Findings
1
Transformer-XL achieves substantially better performance on both short and long sequences and is up to 1,800+ times faster than vanilla Transformers during evaluation
2
Transformer-XL introduces a segment-level recurrence mechanism and a novel positional encoding to enable learning beyond fixed-length context without disrupting temporal coherence
3
Transformer-XL resolves the context fragmentation problem and captures longer-term dependency 80% longer than RNNs and 450% longer than vanilla Transformers
4
Transformer-XL sets new state-of-the-art results (bpc/perplexity): 0.99 on en-wiki8, 1.08 on text8, 18.3 on WikiText-103, 21.8 on One Billion Word, and 54.5 on Penn Treebank (without finetuning)
5
When trained only on WikiText-103, Transformer-XL can generate reasonably coherent, novel text articles with thousands of tokens
Research Object
Transformer-XL neural language model (architecture with segment-level recurrence and novel positional encoding)
Research Subject
Ability to learn and model longer-term dependencies beyond a fixed-length context, resolving context fragmentation, and resulting performance (dependency length learned, bpc/perplexity, generation coherence, and evaluation speed) compared to RNNs and vanilla Transformers
Publication Details
Publication Date
2019-01-01
Journal
Publisher
ISSN
Cited by
3420
Open access PDF
Access Type
Author Information
Download PDF
Subscribe to digest
References available in scid.ai5
Exploiting Generative AI to Scale up Intelligent Tutoring Systems2023
Neural Machine Translation by Jointly Learning to Align and Translate2014
Recurrent neural network based language model2010
Self-Attention with Relative Position Representations2018
Qualitative Spatial Reasoning over Questions (Short Paper)2022
Cited by20
Transformers: State-of-the-Art Natural Language Processing2020
A Survey of Large Language Models2026
Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting2021
Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers2021
Lost in the Middle: How Language Models Use Long Contexts2024
A Comprehensive Overview of Large Language Models2025
Point Transformer2021
Attention mechanisms in computer vision: A survey2022
On the Opportunities and Risks of Foundation Models2021
Longformer: The Long-Document Transformer2020
Transformers in Time Series: A Survey2023
Time-series forecasting with deep learning: a survey2021
Recent Advances in Natural Language Processing via Large Pre-trained Language Models: A Survey2023
Efficient Transformers: A Survey2022
Multimodal Learning With Transformers: A Survey2023
K-BERT: Enabling Language Representation with Knowledge Graph2020
Comprehensive Review of Artificial Neural Network Applications to Pattern Recognition2019
ERNIE 2.0: A Continual Pre-Training Framework for Language Understanding2020
Racial disparities in automated speech recognition2020
Modeling aspects of the language of life through transfer-learning protein sequences2019