Improving Transformer‐Based Neural Machine Translation with Prior Alignments
Open Access
- 1 January 2021
- journal article
- research article
- Published by Wiley in Complexity
- Vol. 2021 (1)
Abstract
Transformer is a neural machine translation model which revolutionizes machine translation. Compared with traditional statistical machine translation models and other neural machine translation models, the recently proposed transformer model radically and fundamentally changes machine translation with its self‐attention and cross‐attention mechanisms. These mechanisms effectively model token alignments between source and target sentences. It has been reported that the transformer model provides accurate posterior alignments. In this work, we empirically prove the reverse effect, showing that prior alignments help transformer models produce better translations. Experiment results on Vietnamese‐English news translation task show not only the positive effect of manually annotated alignments on transformer models but also the surprising outperformance of statistically constructed alignments reinforced with the flexibility of token‐type selection over manual alignments in improving transformer models. Statistically constructed word‐to‐lemma alignments are used to train a word‐to‐word transformer model. The novel hybrid transformer model improves the baseline transformer model and transformer model trained with manual alignments by 2.53 and 0.79 BLEU, respectively. In addition to BLEU score, we make limited human judgment on translation results. Strong correlation between human and machine judgment confirms our findings.Keywords
This publication has 20 references indexed in Scilit:
- Source-Word Decomposition for Neural Machine TranslationMathematical Problems in Engineering, 2020
- Mixed-Level Neural Machine TranslationComputational Intelligence and Neuroscience, 2020
- State-of-the-art augmented NLP transformer models for direct and single-step retrosynthesisNature Communications, 2020
- Explicitly Modeling Word Translations in Neural Machine TranslationACM Transactions on Asian and Low-Resource Language Information Processing, 2019
- Pre-Alignment Guided Attention for Improving Training Efficiency and Model Stability in End-to-End Speech SynthesisIEEE Access, 2019
- Jointly Learning to Align and Translate with Transformer Models2019
- Generating Alignments Using Target Foresight in Attention-Based Neural Machine TranslationThe Prague Bulletin of Mathematical Linguistics, 2017
- Building an English-Vietnamese Bilingual Corpus for Machine Translation2012
- Pharaoh: A Beam Search Decoder for Phrase-Based Statistical Machine Translation Models2004
- A Systematic Comparison of Various Statistical Alignment ModelsComputational Linguistics, 2003