Mitigating Metric Bias in Minimum Bayes Risk Decoding
Authors:
- Geza KovacsGKGeza Kovacs
- Daniel DeutschDDDaniel Deutsch
- Markus FreitagMFMarkus Freitag
Preprint
Abstract
While Minimum Bayes Risk (MBR) decoding using metrics such as COMET or MetricX has outperformed traditional decoding methods such as greedy or beam search, it introduces a challenge we refer to as metric bias. As MBR decoding aims to produce translations that score highly according to a specific utility metric, this very process makes it impossible to use the same metric for both decoding and evaluation, as improvements might simply be due to reward hacking rather than reflecting real quality improvements. In this work we find that compared to human ratings, neural metrics not only overestimate the quality of MBR decoding when the same metric is used as the utility metric, but they also overestimate the quality of MBR/QE decoding with other neural utility metrics as well. We also show that the metric bias issue can be mitigated by using an ensemble of utility metrics during MBR decoding: human evaluations show that MBR decoding using an ensemble of utility metrics outperforms a single utility metric.Keywords
All Related Versions
This publication has 7 references indexed in Scilit:
- Centroid-Based Efficient Minimum Bayes Risk Decoding2024
- NAIST-NICT WMT’23 General MT Task Submission2023
- Faster Minimum Bayes Risk Decoding with Confidence-based Pruning2023
- Epsilon Sampling Rocks: Investigating Sampling Strategies for Minimum Bayes Risk Decoding for Machine Translation2023
- Sampling-Based Approximations to Minimum Bayes Risk Decoding for Neural Machine Translation2022
- Quality-Aware Decoding for Neural Machine Translation2022
- Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine TranslationTransactions of the Association for Computational Linguistics, 2021