Learning Deep Transformer Models for Machine Translation

Обучение глубоких моделей Transformer для машинного перевода
Jingbo Zhu, Qiang Wang, Tong Xiao, Bei Li, Changliang Li, Derek F. Wong, Lidia Sam Chao
2019-06-05

Transformer-Bigdeep Transformerlayer normalizationmachine translationresidual layer aggregation
Transformer is the state-of-the-art model in recent machine translation evaluations. Two strands of research are promising to improve models of this kind: the first uses wide networks (a.k.a. Transformer-Big) and has been the de facto standard for the development of the Transformer system, and the other uses deeper language representation but faces the difficulty arising from learning deep networks. Here, we continue the line of research on the latter. We claim that a truly deep Transformer model can surpass the Transformer-Big counterpart by 1) proper use of layer normalization and 2) a novel way of passing the combination of previous layers to the next. On WMT'16 English- German, NIST OpenMT'12 Chinese-English and larger WMT'18 Chinese-English tasks, our deep system (30/25-layer encoder) outperforms the shallow Transformer-Big/Base baseline (6-layer encoder) by 0.4-2.4 BLEU points. As another bonus, the deep model is 1.6X smaller in size and 3X faster in training than Transformer-Big.
1
A truly deep Transformer with proper layer normalization and a novel method of passing combined previous layers can surpass Transformer-Big.
2
On WMT'16 English-German, NIST OpenMT'12 Chinese-English, and WMT'18 Chinese-English, a 30/25-layer encoder deep system outperforms 6-layer Transformer-Big/Base by 0.4–2.4 BLEU.
3
The proposed deep model is 1.6× smaller in parameter size than Transformer-Big.
4
The proposed deep model trains about 3× faster than Transformer-Big.

Deep Transformer models for machine translation (30/25-layer encoder architectures)

Training and architectural techniques (layer normalization usage and novel layer-combination/passing) to enable learning of very deep Transformer networks and their impact on translation performance, model size, and training speed compared to Transformer-Big/Base

Publication Details
Publication Date
2019-06-05
Journal
Publisher
ISSN
Cited by
96
Access Type
Author Information
Authors
Jingbo Zhu
Qiang Wang
Tong Xiao
Bei Li
Changliang Li
Derek F. Wong
Lidia Sam Chao
Explore further
Open the scid.ai AI chat with a ready-made request: it will find papers on a similar topic and help build a literature review.
Find similar papers in the chat →
Make a presentation
100%