Longformer: The Long-Document Transformer

Longformer: трансформер для длинных документов
Matthew E. Peters, Iz Beltagy, Arman Cohan
2020-04-10

LongformerLongformer-Encoder-Decoder (LED)TriviaQAWikiHoparXiv summarizationcharacter-level language modelingenwik8global attentionlinear-scaling attentionlocal windowed attentionlong-document transformerpretraining and finetuningtext8
Transformer-based models are unable to process long sequences due to their self-attention operation, which scales quadratically with the sequence length. To address this limitation, we introduce the Longformer with an attention mechanism that scales linearly with sequence length, making it easy to process documents of thousands of tokens or longer. Longformer's attention mechanism is a drop-in replacement for the standard self-attention and combines a local windowed attention with a task motivated global attention. Following prior work on long-sequence transformers, we evaluate Longformer on character-level language modeling and achieve state-of-the-art results on text8 and enwik8. In contrast to most prior work, we also pretrain Longformer and finetune it on a variety of downstream tasks. Our pretrained Longformer consistently outperforms RoBERTa on long document tasks and sets new state-of-the-art results on WikiHop and TriviaQA. We finally introduce the Longformer-Encoder-Decoder (LED), a Longformer variant for supporting long document generative sequence-to-sequence tasks, and demonstrate its effectiveness on the arXiv summarization dataset.
1
A pretrained Longformer when fine-tuned consistently outperforms RoBERTa on long-document tasks.
2
Longformer introduces an attention mechanism that scales linearly with sequence length, enabling processing of documents thousands of tokens long.
3
Longformer sets new state-of-the-art results on the QA benchmarks WikiHop and TriviaQA.
4
Longformer’s attention is a drop-in replacement combining local windowed attention with task-motivated global attention.
5
On character-level language modeling, Longformer achieves state-of-the-art results on text8 and enwik8.
6
The paper introduces Longformer-Encoder-Decoder (LED) for long-document sequence-to-sequence generation and demonstrates its effectiveness on the arXiv summarization dataset.

Longformer model (a Transformer variant with linear-scaling attention for long documents)

The attention mechanism and model performance for processing long sequences, including a combined local-window and task-motivated global attention that enables linear scaling and improves downstream long-document tasks (pretraining, fine-tuning, QA, summarization)

Publication Details
Publication Date
2020-04-10
Journal
Publisher
ISSN
Cited by
2281
Access Type
Author Information
Authors
Matthew E. Peters
Iz Beltagy
Arman Cohan
Explore further
Open the scid.ai AI chat with a ready-made request: it will find papers on a similar topic and help build a literature review.
Find similar papers in the chat →
Make a presentation
100%