On The Evaluation of Machine Translation Systems Trained With Back-Translation
Authors:
- Sergey EdunovSESergey Edunov
- Myle OttMOMyle Ott
- Marc'Aurelio RanzatoMRMarc'Aurelio Ranzato
Preprint
Abstract
Back-translation is a widely used data augmentation technique which leverages target monolingual data. However, its effectiveness has been challenged since automatic metrics such as BLEU only show significant improvements for test examples where the source itself is a translation, or translationese. This is believed to be due to translationese inputs better matching the back-translated training data. In this work, we show that this conjecture is not empirically supported and that back-translation improves translation quality of both naturally occurring text as well as translationese according to professional human translators. We provide empirical evidence to support the view that back-translation is preferred by humans because it produces more fluent outputs. BLEU cannot capture human preferences because references are translationese when source sentences are natural text. We recommend complementing BLEU with a language model score to measure fluency.Keywords
All Related Versions
This publication has 7 references indexed in Scilit:
- Statistical Power and Translationese in Machine Translation Evaluation2020
- Findings of the 2019 Conference on Machine Translation (WMT19)2019
- The Effect of Translationese in Machine Translation Test Sets2019
- Phrase-Based & Neural Unsupervised Machine Translation2018
- Attaining the Unattainable? Reassessing Claims of Human Parity in Neural Machine Translation2018
- Findings of the 2018 Conference on Machine Translation (WMT18)2018
- Descriptive translation studies and beyond . By Gideon Toury. Amsterdam: John Benjamins, 1995. Pp. viii, 312. Cloth $84.00, paper $27.95.Language, 1997