Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
Authors:
Preprint
Abstract
Machine translation is a natural candidate problem for reinforcement learning from human feedback: users provide quick, dirty ratings on candidate translations to guide a system to improve. Yet, current neural machine translation training focuses on expensive human-generated reference translations. We describe a reinforcement learning algorithm that improves neural machine translation systems from simulated human feedback. Our algorithm combines the advantage actor-critic algorithm (Mnih et al., 2016) with the attention-based neural encoder-decoder architecture (Luong et al., 2015). This algorithm (a) is well-designed for problems with a large action space and delayed rewards, (b) effectively optimizes traditional corpus-level machine translation metrics, and (c) is robust to skewed, high-variance, granular feedback modeled after actual human behaviors.Keywords
All Related Versions
This publication has 5 references indexed in Scilit:
- Personalized Machine Translation: Preserving Original Author Traits2017
- Can machine translation systems be evaluated by the crowd aloneNatural Language Engineering, 2015
- Crowdsourced Monolingual TranslationACM Transactions on Computer-Human Interaction, 2014
- Impact of data characteristics on recommender systems performanceACM Transactions on Management Information Systems, 2012
- Active learning in heteroscedastic noiseTheoretical Computer Science, 2010