COMET: A Neural Framework for MT Evaluation
Preprint
Abstract
We present COMET, a neural framework for training multilingual machine translation evaluation models which obtains new state-of-the-art levels of correlation with human judgements. Our framework leverages recent breakthroughs in cross-lingual pretrained language modeling resulting in highly multilingual and adaptable MT evaluation models that exploit information from both the source input and a target-language reference translation in order to more accurately predict MT quality. To showcase our framework, we train three models with different types of human judgements: Direct Assessments, Human-mediated Translation Edit Rate and Multidimensional Quality Metrics. Our models achieve new state-of-the-art performance on the WMT 2019 Metrics shared task and demonstrate robustness to high-performing systems.Keywords
All Related Versions
This publication has 38 references indexed in Scilit:
- On the Limitations of Cross-lingual Encoders as Exposed by Reference-Free Machine Translation Evaluation2020
- Unsupervised Cross-lingual Representation Learning at Scale2020
- BLEURT: Learning Robust Metrics for Text Generation2020
- Automatic Machine Translation Evaluation using Source Language Inputs and Cross-lingual Language Model2020
- Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and BeyondTransactions of the Association for Computational Linguistics, 2019
- Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks2019
- 75 Languages, 1 Model: Parsing Universal Dependencies Universally2019
- MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance2019
- Can machine translation systems be evaluated by the crowd aloneNatural Language Engineering, 2015
- The Meteor metric for automatic evaluation of machine translationMachine Translation, 2009