Why word error rate is not a good metric for speech recognizer training for the speech translation task?
Authors:
- 1 May 2011
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 5632-5635
Abstract
Speech translation (ST) is an enabling technology for cross-lingual oral communication. A ST system consists of two major components: an automatic speech recognizer (ASR) and a machine translator (MT). Nowadays, most ASR systems are trained and tuned by minimizing word error rate (WER). However, WER counts word errors at the surface level. It does not consider the contextual and syntactic roles of a word, which are often critical for MT. In the end-to-end ST scenarios, whether WER is a good metric for the ASR component of the full ST system is an open issue and lacks systematic studies. In this paper, we report our recent investigation on this issue, focusing on the interactions of ASR and MT in a ST system. We show that BLEU-oriented global optimization of ASR system parameters improves the translation quality by an absolute 1.5% BLEU score, while sacrificing WER over the conventional, WER-optimized ASR system. We also conducted an in-depth study on the impact of ASR errors on the final ST output. Our findings suggest that the speech recognizer component of the full ST system should be optimized by translation metrics instead of the traditional WER.Keywords
This publication has 13 references indexed in Scilit:
- A novel decision function and the associated decision-feedback learning for speech translation2011
- Automatic disfluency removal for improving spoken language translation2010
- An Integrative and Discriminative Technique for Spoken Utterance ClassificationIEEE Transactions on Audio, Speech, and Language Processing, 2008
- Recent efforts in spoken language translationIEEE Signal Processing Magazine, 2008
- Integrating Speech Recognition and Machine Translation2007
- A hierarchical phrase-based model for statistical machine translation2005
- Phrase-based translation of speech recognizer word lattices using loglinear model combination2005
- A unified approach in speech-to-speech translation2004
- Statistical phrase-based translation2003
- Speech translation: coupling of recognition and translation1999