Self-Attention with Relative Position Representations

Механизм самовнимания с представлениями относительной позиции
Ashish Vaswani, Jakob Uszkoreit, Peter Shaw
2018-01-01
SCID:  54.1/ng8mdjd6 Discuss with AI
Attention MechanismRelative Position RepresentationsSelf-AttentionShaw Uszkoreit VaswaniTransformers
Abstract (AI)
Peter Shaw, Jakob Uszkoreit, Ashish Vaswani. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers). 2018.
Key Findings
1
Augmenting self-attention with relative position information allows the model to better capture sequence order and distances between tokens.
2
Introducing relative position representations for self-attention improves modeling of token interactions compared to absolute positional encodings.
3
The proposed relative position representations can be integrated into the transformer-style self-attention mechanism without major architectural changes.
4
Using relative position representations leads to improved empirical performance on tasks evaluated in the paper (reported in the 2018 NAACL short paper).
Research Object

Self-attention mechanism in neural sequence models

Research Subject

Incorporation of relative position representations into self-attention to model positional relationships between sequence elements

Publication Details
Publication Date
2018-01-01
Journal
Publisher
ISSN
Access Type
Author Information
Authors
Ashish Vaswani
Jakob Uszkoreit
Peter Shaw
Explore further
Open the scid.ai AI chat with a ready-made request: it will find papers on a similar topic and help build a literature review.
Find similar papers in the chat →
Make a presentation
100%