Infusing Sequential Information into Conditional Masked Translation Model with Self-Review Mechanism
Preprint
Abstract
Non-autoregressive models generate target words in a parallel way, which achieve a faster decoding speed but at the sacrifice of translation accuracy. To remedy a flawed translation by non-autoregressive models, a promising approach is to train a conditional masked translation model (CMTM), and refine the generated results within several iterations. Unfortunately, such approach hardly considers the \textit{sequential dependency} among target words, which inevitably results in a translation degradation. Hence, instead of solely training a Transformer-based CMTM, we propose a Self-Review Mechanism to infuse sequential information into it. Concretely, we insert a left-to-right mask to the same decoder of CMTM, and then induce it to autoregressively review whether each generated word from CMTM is supposed to be replaced or kept. The experimental results (WMT14 En$\leftrightarrow$De and WMT16 En$\leftrightarrow$Ro) demonstrate that our model uses dramatically less training computations than the typical CMTM, as well as outperforms several state-of-the-art non-autoregressive models by over 1 BLEU. Through knowledge distillation, our model even surpasses a typical left-to-right Transformer model, while significantly speeding up decoding.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- A Study of Non-autoregressive Model for Sequence Generation2020
- Non-Autoregressive Machine Translation with Auxiliary RegularizationProceedings of the AAAI Conference on Artificial Intelligence, 2019
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding2019
- Mask-Predict: Parallel Decoding of Conditional Masked Language Models2019
- Hint-Based Training for Non-Autoregressive Machine Translation2019
- FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flow2019
- Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement2018
- Neural Machine Translation of Rare Words with Subword Units2016
- Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation2014