Fluent and Low-latency Simultaneous Speech-to-Speech Translation with Self-adaptive Training
Preprint
Abstract
Simultaneous speech-to-speech translation is widely useful but extremely challenging, since it needs to generate target-language speech concurrently with the source-language speech, with only a few seconds delay. In addition, it needs to continuously translate a stream of sentences, but all recent solutions merely focus on the single-sentence scenario. As a result, current approaches accumulate latencies progressively when the speaker talks faster, and introduce unnatural pauses when the speaker talks slower. To overcome these issues, we propose Self-Adaptive Translation (SAT) which flexibly adjusts the length of translations to accommodate different source speech rates. At similar levels of translation quality (as measured by BLEU), our method generates more fluent target speech (as measured by the naturalness metric MOS) with substantially lower latency than the baseline, in both Zh <-> En directions.Keywords
All Related Versions
This publication has 14 references indexed in Scilit:
- Minimum Latency Training Strategies for Streaming Sequence-to-Sequence ASR2020
- A Streaming On-Device End-To-End Model Surpassing Server-Side Conventional Model Quality and Latency2020
- Re-Translation Strategies for Long Form, Simultaneous, Spoken Language Translation2020
- Towards Fast and Accurate Streaming End-To-End ASR2020
- Towards Multimodal Simultaneous Neural Machine Translation2020
- Opportunistic Decoding with Timely Correction for Simultaneous Translation2020
- Natural TTS Synthesis by Conditioning Wavenet on MEL Spectrogram Predictions2018
- Neural Machine Translation of Rare Words with Subword Units2016
- Recognition of time-compressed speech does not predict recognition of natural fast-rate speech by older listenersThe Journal of the Acoustical Society of America, 2014
- Speaker identification on the SCOTUS corpusThe Journal of the Acoustical Society of America, 2008