Effective Approaches to Attention-based Neural Machine Translation
Эффективные подходы к нейронному машинному переводу на основе механизма внимания
2015-01-01
SCID: 54.1/qtd5666f
Discuss with AI
BLEUWMT English-Germanattention-based neural machine translationglobal attentionlocal attention
Figures from the paper
Abstract (AI)
An attentional mechanism has lately been used to improve neural machine translation (NMT) by selectively focusing on parts of the source sentence during translation.However, there has been little work exploring useful architectures for attention-based NMT.This paper examines two simple and effective classes of attentional mechanism: a global approach which always attends to all source words and a local one that only looks at a subset of source words at a time.We demonstrate the effectiveness of both approaches on the WMT translation tasks between English and German in both directions.With local attention, we achieve a significant gain of 5.0 BLEU points over non-attentional systems that already incorporate known techniques such as dropout.Our ensemble model using different attention architectures yields a new state-of-the-art result in the WMT'15 English to German translation task with 25.9 BLEU points, an improvement of 1.0 BLEU points over the existing best system backed by NMT and an n-gram reranker.1
Key Findings
1
An ensemble combining different attention architectures achieves a new WMT'15 English→German state-of-the-art of 25.9 BLEU, improving by 1.0 BLEU over the previous best NMT plus n-gram reranker system.
2
Both global and local attention approaches are effective on WMT English↔German translation tasks.
3
Local attention yields a significant gain of 5.0 BLEU points over non-attentional systems that include techniques like dropout.
4
Two simple attentional mechanisms for NMT are examined: a global approach attending to all source words and a local approach attending to a subset of source words.
Research Object
Attention-based neural machine translation models (global and local attention architectures) applied to English–German WMT translation tasks
Research Subject
Effectiveness and performance of global versus local attention mechanisms, measured by BLEU improvements and state-of-the-art results on English–German WMT translation tasks
Publication Details
Publication Date
2015-01-01
Journal
Publisher
ISSN
Cited by
8382
Open access PDF
Access Type
Author Information
Download PDF
Subscribe to digest
References available in scid.ai3
Cited by19
A Comprehensive Survey on Graph Neural Networks2020
Self-Attention with Relative Position Representations2018
Google’s Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation2017
Deep learning in bioinformatics2016
Earthquake transformer—an attentive deep-learning model for simultaneous earthquake detection and phase picking2020
A Survey on the Explainability of Supervised Machine Learning2021
Recurrent Neural Networks: A Comprehensive Review of Architectures, Variants, and Applications2024
Transfer Learning for Low-Resource Neural Machine Translation2016
Sequence-Level Knowledge Distillation2016
Training Spiking Neural Networks Using Lessons From Deep Learning2023
Attention in Natural Language Processing2020
Learning Deep Transformer Models for Machine Translation2019
Modeling Coverage for Neural Machine Translation2016
Semantic Communications for Future Internet: Fundamentals, Applications, and Challenges2022
DeePMD-kit v2: A software package for deep potential models2023
Adaptation of Deep Bidirectional Multilingual Transformers for Russian Language2019
Large language models (LLMs): survey, technical frameworks, and future challenges2024
Graph Neural Networks for Natural Language Processing: A Survey2023
Anomaly Detection in Dynamic Graphs: A Comprehensive Survey2024