Online Versus Offline NMT Quality: An In-depth Analysis on English-German and German-English
Authors:
Preprint
Abstract
We conduct in this work an evaluation study comparing offline and online neural machine translation architectures. Two sequence-to-sequence models: convolutional Pervasive Attention (Elbayad et al. 2018) and attention-based Transformer (Vaswani et al. 2017) are considered. We investigate, for both architectures, the impact of online decoding constraints on the translation quality through a carefully designed human evaluation on English-German and German-English language pairs, the latter being particularly sensitive to latency constraints. The evaluation results allow us to identify the strengths and shortcomings of each model when we shift to the online setup.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- Low-Latency Sequence-to-Sequence Speech Recognition and Translation by Partial Hypothesis Selection2020
- A Set of Recommendations for Assessing Human–Machine Parity in Language TranslationJournal of Artificial Intelligence Research, 2020
- STACL: Simultaneous Translation with Implicit Anticipation and Controllable Latency using Prefix-to-Prefix Framework2019
- Findings of the 2019 Conference on Machine Translation (WMT19)2019
- Quantitative fine-grained human evaluation of machine translation systems: a case study on English to CroatianMachine Translation, 2018
- Incremental Decoding and Training Methods for Simultaneous Translation in Neural Machine Translation2018
- Don’t Until the Final Verb Wait: Reinforcement Learning for Simultaneous Machine Translation2014
- Towards Automatic Error Analysis of Machine Translation OutputComputational Linguistics, 2011
- Simultaneous translation of lectures and speechesMachine Translation, 2007
- A Coefficient of Agreement for Nominal ScalesEducational and Psychological Measurement, 1960