Transformer-XL: Attentive Language Models beyond a Fixed-Length Context

Transformer-XL: внимательные языковые модели за пределами контекста фиксированной длины
Quoc Viet Le, Yiming Yang, Zihang Dai, Ruslan Salakhutdinov, Zhilin Yang, Jaime Carbonell, Zihang Dai
2019-01-01

Transformer-XLcontext fragmentationlong-range dependencynovel positional encodingsegment-level recurrence
Transformers have a potential of learning longer-term dependency, but are limited by a fixed-length context in the setting of language modeling.We propose a novel neural architecture Transformer-XL that enables learning dependency beyond a fixed length without disrupting temporal coherence.It consists of a segment-level recurrence mechanism and a novel positional encoding scheme.Our method not only enables capturing longer-term dependency, but also resolves the context fragmentation problem.As a result, Transformer-XL learns dependency that is 80% longer than RNNs and 450% longer than vanilla Transformers, achieves better performance on both short and long sequences, and is up to 1,800+ times faster than vanilla Transformers during evaluation.Notably, we improve the state-ofthe-art results of bpc/perplexity to 0.99 on en-wiki8, 1.08 on text8, 18.3 on WikiText-103, 21.8 on One Billion Word, and 54.5 on Penn Treebank (without finetuning).When trained only on WikiText-103, Transformer-XL manages to generate reasonably coherent, novel text articles with thousands of tokens.Our code, pretrained models, and hyperparameters are available in both Tensorflow and PyTorch 1 .
1
Transformer-XL achieves substantially better performance on both short and long sequences and is up to 1,800+ times faster than vanilla Transformers during evaluation
2
Transformer-XL introduces a segment-level recurrence mechanism and a novel positional encoding to enable learning beyond fixed-length context without disrupting temporal coherence
3
Transformer-XL resolves the context fragmentation problem and captures longer-term dependency 80% longer than RNNs and 450% longer than vanilla Transformers
4
Transformer-XL sets new state-of-the-art results (bpc/perplexity): 0.99 on en-wiki8, 1.08 on text8, 18.3 on WikiText-103, 21.8 on One Billion Word, and 54.5 on Penn Treebank (without finetuning)
5
When trained only on WikiText-103, Transformer-XL can generate reasonably coherent, novel text articles with thousands of tokens

Transformer-XL neural language model (architecture with segment-level recurrence and novel positional encoding)

Ability to learn and model longer-term dependencies beyond a fixed-length context, resolving context fragmentation, and resulting performance (dependency length learned, bpc/perplexity, generation coherence, and evaluation speed) compared to RNNs and vanilla Transformers

Publication Details
Publication Date
2019-01-01
Journal
Publisher
ISSN
Cited by
3420
Access Type
Author Information
Authors
Quoc Viet Le
Yiming Yang
Zihang Dai
Ruslan Salakhutdinov
Zhilin Yang
Jaime Carbonell
Zihang Dai
Explore further
Open the scid.ai AI chat with a ready-made request: it will find papers on a similar topic and help build a literature review.
Find similar papers in the chat →
Make a presentation
100%