The FLoRes Evaluation Datasets for Low-Resource Machine Translation: Nepali-English and Sinhala-English
Authors:
- Philipp KoehnPKPhilipp Koehn
- Marc'Aurelio RanzatoMRMarc'Aurelio Ranzato
Preprint
Abstract
For machine translation, a vast majority of language pairs in the world are considered low-resource because they have little parallel data available. Besides the technical challenges of learning with limited supervision, it is difficult to evaluate methods trained on low-resource language pairs because of the lack of freely and publicly available benchmarks. In this work, we introduce the FLoRes evaluation datasets for Nepali-English and Sinhala-English, based on sentences translated from Wikipedia. Compared to English, these are languages with very different morphology and syntax, for which little out-of-domain parallel data is available and for which relatively large amounts of monolingual data are freely available. We describe our process to collect and cross-check the quality of translations, and we report baseline performance using several learning settings: fully supervised, weakly supervised, semi-supervised, and fully unsupervised. Our experiments demonstrate that current state-of-the-art methods perform rather poorly on this benchmark, posing a challenge to the research community working on low-resource MT. Data and code to reproduce our experiments are available at https://github.com/facebookresearch/flores.Keywords
All Related Versions
This publication has 7 references indexed in Scilit:
- Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and BeyondTransactions of the Association for Computational Linguistics, 2019
- A Call for Clarity in Reporting BLEU Scores2018
- Google’s Multilingual Neural Machine Translation System: Enabling Zero-Shot TranslationTransactions of the Association for Computational Linguistics, 2017
- Enriching Word Vectors with Subword InformationTransactions of the Association for Computational Linguistics, 2017
- On integrating a language model into neural machine translationComputer Speech & Language, 2017
- A Convolutional Encoder Model for Neural Machine Translation2017
- A New Approach to the Study of Translationese: Machine-learning the Difference between Original and Translated TextLiterary and Linguistic Computing, 2005