Consistent Transcription and Translation of Speech
Authors:
- Matthias SperberMSMatthias Sperber
- Hendra SetiawanHSHendra Setiawan
- Christian GollanCGChristian Gollan
- Udhyakumar NallasamyUNUdhyakumar Nallasamy
- Matthias PaulikMPMatthias Paulik
Preprint
Abstract
The conventional paradigm in speech translation starts with a speech recognition step to generate transcripts, followed by a translation step with the automatic transcripts as input. To address various shortcomings of this paradigm, recent work explores end-to-end trainable direct models that translate without transcribing. However, transcripts can be an indispensable output in practical applications, which often display transcripts alongside the translations to users. We make this common requirement explicit and explore the task of jointly transcribing and translating speech. While high accuracy of transcript and translation are crucial, even highly accurate systems can suffer from inconsistencies between both outputs that degrade the user experience. We introduce a methodology to evaluate consistency and compare several modeling approaches, including the traditional cascaded approach and end-to-end models. We find that direct models are poorly suited to the joint transcription/translation task, but that end-to-end models that feature a coupled inference procedure are able to achieve strong consistency. We further introduce simple techniques for directly optimizing for consistency, and analyze the resulting trade-offs between consistency, transcription accuracy, and translation accuracy.Keywords
All Related Versions
This publication has 13 references indexed in Scilit:
- Attention-Passing Models for Robust and Data-Efficient End-to-End Speech TranslationTransactions of the Association for Computational Linguistics, 2019
- Are Red Roses Red? Evaluating Consistency of Question-Answering Models2019
- Findings of the WMT 2019 Shared Tasks on Quality Estimation2019
- SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing2018
- A Call for Clarity in Reporting BLEU Scores2018
- Speech Recognition, Machine Translation, and Speech Translation—A Unified Discriminative Learning Paradigm [Lecture Notes]IEEE Signal Processing Magazine, 2011
- Using Mechanical Turk to Obtain and Analyze English Acceptability JudgmentsLanguage and Linguistics Compass, 2011
- Why word error rate is not a good metric for speech recognizer training for the speech translation task?2011
- Likert scales, levels of measurement and the “laws” of statisticsAdvances in Health Sciences Education, 2010
- JANUS: a speech-to-speech translation system using connectionist and symbolic processing strategies1991