Transfer Learning for Low-Resource Neural Machine Translation

Трансферное обучение для нейронного машинного перевода при ограниченных ресурсах
Barret Zoph, Kevin K. Knight, Jonathan May, Deniz Yüret
2016-01-01

BLEU improvementlow-resource languagesneural machine translationsyntax based machine translationtransfer learning
The encoder-decoder framework for neural machine translation (NMT) has been shown effective in large data scenarios, but is much less effective for low-resource languages.We present a transfer learning method that significantly improves BLEU scores across a range of low-resource languages.Our key idea is to first train a high-resource language pair (the parent model), then transfer some of the learned parameters to the low-resource pair (the child model) to initialize and constrain training.Using our transfer learning method we improve baseline NMT models by an average of 5.6 BLEU on four low-resource language pairs.Ensembling and unknown word replacement add another 2 BLEU which brings the NMT performance on low-resource machine translation close to a strong syntax based machine translation (SBMT) system, exceeding its performance on one language pair.Additionally, using the transfer learning model for re-scoring, we can improve the SBMT system by an average of 1.3 BLEU, improving the state-of-the-art on low-resource machine translation.
1
A transfer learning method for NMT trains a high-resource parent model then transfers parameters to initialize and constrain a low-resource child model.
2
Ensembling and unknown word replacement add about 2 BLEU, bringing NMT performance close to a strong syntax-based MT (SBMT) system and exceeding it on one language pair.
3
The transfer learning approach improves baseline NMT models by an average of 5.6 BLEU across four low-resource language pairs.
4
Using the transfer-learned NMT model to re-score SBMT outputs improves the SBMT system by an average of 1.3 BLEU, advancing state-of-the-art for low-resource MT.

Transfer learning method for low-resource neural machine translation (parent-to-child parameter transfer between high-resource and low-resource language pairs)

Improvement of NMT performance (BLEU score gains and bringing low-resource NMT close to or exceeding syntax-based MT) via parameter transfer initialization/constraint, ensembling, unknown-word replacement, and rescoring

Publication Details
Publication Date
2016-01-01
Journal
Publisher
ISSN
Cited by
918
Access Type
Author Information
Authors
Barret Zoph
Kevin K. Knight
Jonathan May
Deniz Yüret
Explore further
Open the scid.ai AI chat with a ready-made request: it will find papers on a similar topic and help build a literature review.
Find similar papers in the chat →
Make a presentation
100%