MetricX-24: The Google Submission to the WMT 2024 Metrics Shared Task
Authors:
- Juraj JuraskaJJJuraj Juraska
- Daniel DeutschDDDaniel Deutsch
- Mara FinkelsteinMFMara Finkelstein
- Markus FreitagMFMarkus Freitag
Preprint
Abstract
In this paper, we present the MetricX-24 submissions to the WMT24 Metrics Shared Task and provide details on the improvements we made over the previous version of MetricX. Our primary submission is a hybrid reference-based/-free metric, which can score a translation irrespective of whether it is given the source segment, the reference, or both. The metric is trained on previous WMT data in a two-stage fashion, first on the DA ratings only, then on a mixture of MQM and DA ratings. The training set in both stages is augmented with synthetic examples that we created to make the metric more robust to several common failure modes, such as fluent but unrelated translation, or undertranslation. We demonstrate the benefits of the individual modifications via an ablation study, and show a significant performance increase over MetricX-23 on the WMT23 MQM ratings, as well as our new synthetic challenge set.Keywords
All Related Versions
This publication has 18 references indexed in Scilit:
- The Eval4NLP 2023 Shared Task on Prompting Large Language Models as Explainable Metrics2023
- INSTRUCTSCORE: Towards Explainable Text Generation Evaluation with Automatic Feedback2023
- Ties Matter: Meta-Evaluating Modern Metrics with Pairwise Accuracy and Tie Calibration2023
- DEMETR: Diagnosing Evaluation Metrics for Translation2022
- Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine TranslationTransactions of the Association for Computational Linguistics, 2021
- Learning Compact Metrics for MT2021
- mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer2021
- COMET: A Neural Framework for MT Evaluation2020
- Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasing2020
- BLEURT: Learning Robust Metrics for Text Generation2020