Longformer: The Long-Document Transformer
Longformer: трансформер для длинных документов
2020-04-10
SCID: 54.1/ppds75bs
Discuss with AI
LongformerLongformer-Encoder-Decoder (LED)TriviaQAWikiHoparXiv summarizationcharacter-level language modelingenwik8global attentionlinear-scaling attentionlocal windowed attentionlong-document transformerpretraining and finetuningtext8
Figures from the paper
Abstract (AI)
Transformer-based models are unable to process long sequences due to their self-attention operation, which scales quadratically with the sequence length. To address this limitation, we introduce the Longformer with an attention mechanism that scales linearly with sequence length, making it easy to process documents of thousands of tokens or longer. Longformer's attention mechanism is a drop-in replacement for the standard self-attention and combines a local windowed attention with a task motivated global attention. Following prior work on long-sequence transformers, we evaluate Longformer on character-level language modeling and achieve state-of-the-art results on text8 and enwik8. In contrast to most prior work, we also pretrain Longformer and finetune it on a variety of downstream tasks. Our pretrained Longformer consistently outperforms RoBERTa on long document tasks and sets new state-of-the-art results on WikiHop and TriviaQA. We finally introduce the Longformer-Encoder-Decoder (LED), a Longformer variant for supporting long document generative sequence-to-sequence tasks, and demonstrate its effectiveness on the arXiv summarization dataset.
Key Findings
1
A pretrained Longformer when fine-tuned consistently outperforms RoBERTa on long-document tasks.
2
Longformer introduces an attention mechanism that scales linearly with sequence length, enabling processing of documents thousands of tokens long.
3
Longformer sets new state-of-the-art results on the QA benchmarks WikiHop and TriviaQA.
4
Longformer’s attention is a drop-in replacement combining local windowed attention with task-motivated global attention.
5
On character-level language modeling, Longformer achieves state-of-the-art results on text8 and enwik8.
6
The paper introduces Longformer-Encoder-Decoder (LED) for long-document sequence-to-sequence generation and demonstrates its effectiveness on the arXiv summarization dataset.
Research Object
Longformer model (a Transformer variant with linear-scaling attention for long documents)
Research Subject
The attention mechanism and model performance for processing long sequences, including a combined local-window and task-motivated global attention that enables linear scaling and improves downstream long-document tasks (pretraining, fine-tuning, QA, summarization)
Publication Details
Publication Date
2020-04-10
Journal
Publisher
ISSN
Cited by
2281
Open access PDF
Access Type
Author Information
Download PDF
Subscribe to digest
References available in scid.ai3
Cited by17
Transformers: State-of-the-Art Natural Language Processing2020
A Survey of Large Language Models2026
Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting2021
Lost in the Middle: How Language Models Use Long Contexts2024
HuggingFace's Transformers: State-of-the-art Natural Language Processing2019
On the Opportunities and Risks of Foundation Models2021
Unifying Large Language Models and Knowledge Graphs: A Roadmap2024
CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows2022
Efficient Transformers: A Survey2022
Multimodal Learning With Transformers: A Survey2023
Can Large Language Models Transform Computational Social Science?2023
ETC: Encoding Long and Structured Inputs in Transformers2020
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization2022
GENA-LM: a family of open-source foundational DNA language models for long sequences2024
Answering Complex Open-Domain Questions with Multi-Hop Dense Retrieval2020
Materials science in the era of large language models: a perspective2024
SeDR: Segment Representation Learning for Long Documents Dense Retrieval2022