Self-Attention with Relative Position Representations
Механизм самовнимания с представлениями относительной позиции
2018-01-01
SCID: 54.1/ng8mdjd6
Discuss with AI
Attention MechanismRelative Position RepresentationsSelf-AttentionShaw Uszkoreit VaswaniTransformers
Figures from the paper
Abstract (AI)
Peter Shaw, Jakob Uszkoreit, Ashish Vaswani. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers). 2018.
Key Findings
1
Augmenting self-attention with relative position information allows the model to better capture sequence order and distances between tokens.
2
Introducing relative position representations for self-attention improves modeling of token interactions compared to absolute positional encodings.
3
The proposed relative position representations can be integrated into the transformer-style self-attention mechanism without major architectural changes.
4
Using relative position representations leads to improved empirical performance on tasks evaluated in the paper (reported in the 2018 NAACL short paper).
Research Object
Self-attention mechanism in neural sequence models
Research Subject
Incorporation of relative position representations into self-attention to model positional relationships between sequence elements
Publication Details
Publication Date
2018-01-01
Journal
Publisher
ISSN
Cited by
2504
Open access PDF
Access Type
Author Information
Download PDF
Subscribe to digest
References available in scid.ai7
Adam: A Method for Stochastic Optimization2014
Rethinking the Inception Architecture for Computer Vision2016
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation2014
Sequence to Sequence Learning with Neural Networks2014
Effective Approaches to Attention-based Neural Machine Translation2015
Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation2014
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation2016
Cited by16
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer2019
A Survey of Large Language Models2026
Transformer-XL: Attentive Language Models beyond a Fixed-Length Context2019
Attention mechanisms in computer vision: A survey2022
Heterogeneous Graph Transformer2020
Uformer: A General U-Shaped Transformer for Image Restoration2022
Transformers in Time Series: A Survey2023
CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows2022
Learning Deep Transformer Models for Machine Translation2019
Large language models (LLMs): survey, technical frameworks, and future challenges2024
ETC: Encoding Long and Structured Inputs in Transformers2020
Graph Neural Networks for Natural Language Processing: A Survey2023
Human and AI Ensembles: When Can They Work?2021
Large Language Models: A Comprehensive Survey of its Applications, Challenges, Limitations, and Future Prospects2023
A comparative atlas of single-cell chromatin accessibility in the human brain2023
Anomaly Detection in Dynamic Graphs: A Comprehensive Survey2024