Inference Strategies for Machine Translation with Conditional Masking
Preprint
Abstract
Conditional masked language model (CMLM) training has proven successful for non-autoregressive and semi-autoregressive sequence generation tasks, such as machine translation. Given a trained CMLM, however, it is not clear what the best inference strategy is. We formulate masked inference as a factorization of conditional probabilities of partial sequences, show that this does not harm performance, and investigate a number of simple heuristics motivated by this perspective. We identify a thresholding strategy that has advantages over the standard "mask-predict" algorithm, and provide analyses of its behavior on machine translation tasks.Keywords
All Related Versions
This publication has 5 references indexed in Scilit:
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding2019
- Mask-Predict: Parallel Decoding of Conditional Masked Language Models2019
- Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement2018
- A Call for Clarity in Reporting BLEU Scores2018
- Learning What's Easy: Fully Differentiable Neural Easy-First Taggers2017