Improving Machine Translation with Human Feedback: An Exploration of Quality Estimation as a Reward Model
Preprint
Abstract
Insufficient modeling of human preferences within the reward model is a major obstacle for leveraging human feedback to improve translation quality. Fortunately, quality estimation (QE), which predicts the quality of a given translation without reference, has achieved impressive alignment with human evaluations in the last two years. In this work, we investigate the potential of employing the QE model as the reward model (the QE-based reward model) to predict human preferences for feedback training. We first identify the overoptimization problem during QE-based feedback training, manifested as an increase in reward while translation quality declines. We examine the problem and argue that the vulnerability of the QE model might lead to high rewards for incorrect translations, resulting in overoptimization and error propagation. To address the problem, we adopt a simple yet effective method that uses heuristic rules to detect the incorrect translations and assigns a penalty term to the QE-based rewards for the detected incorrect translations. Experimental results show that the proposed QE-based feedback training achieves consistent and significant improvements across various settings, further verified through human preference studies. Our subsequent analysis demonstrates the high data efficiency of the proposed QE-based feedback training: the proposed approach using a small amount of monolingual data can outperform systems using larger parallel corpora.Keywords
All Related Versions
This publication has 19 references indexed in Scilit:
- The Flores-101 Evaluation Benchmark for Low-Resource and Multilingual Machine TranslationTransactions of the Association for Computational Linguistics, 2022
- One Country, 700+ Languages: NLP Challenges for Underrepresented Languages and Dialects in Indonesia2022
- UniTE: Unified Translation Evaluation2022
- WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia2021
- CCMatrix: Mining Billions of High-Quality Parallel Sentences on the Web2021
- Unsupervised Cross-lingual Representation Learning at Scale2020
- BLEURT: Learning Robust Metrics for Text Generation2020
- A Study of Reinforcement Learning for Neural Machine Translation2018
- Counterfactual Learning from Bandit Feedback under Deterministic Logging : A Case Study in Statistical Machine Translation2017
- Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback2017