Minimum Latency Training Strategies for Streaming Sequence-to-Sequence ASR
Authors:
- 1 May 2020
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 6064-6068
Abstract
Recently, a few novel streaming attention-based sequence-to-sequence (S2S) models have been proposed to perform online speech recognition with linear-time decoding complexity. However, in these models, the decisions to generate tokens are delayed compared to the actual acoustic boundaries since their unidirectional encoders lack future information. This leads to an inevitable latency during inference. To alleviate this issue and reduce latency, we propose several strategies during training by leveraging external hard alignments extracted from the hybrid model. We investigate to utilize the alignments in both the encoder and the decoder. On the encoder side, (1) multi-task learning and (2) pre-training with the framewise classification task are studied. On the decoder side, we (3) remove inappropriate alignment paths beyond an acceptable latency during the alignment marginalization, and (4) directly min-imize the differentiable expected latency loss. Experiments on the Cortana voice search task demonstrate that our proposed methods can significantly reduce the latency, and even improve the recognition accuracy in certain cases on the decoder side. We also present some analysis to understand the behaviors of streaming S2S models.Keywords
All Related Versions
This publication has 24 references indexed in Scilit:
- Hybrid CTC/Attention Architecture for End-to-End Speech RecognitionIEEE Journal of Selected Topics in Signal Processing, 2017
- A Comparison of Sequence-to-Sequence Models for Speech Recognition2017
- Multitask Learning with Low-Level Auxiliary Tasks for Encoder-Decoder Based Speech Recognition2017
- Gaussian Prediction Based Attention for Online End-to-End Speech Recognition2017
- On Online Attention-Based Speech Recognition and Joint Mandarin Character-Pinyin Training2016
- Rethinking the Inception Architecture for Computer Vision2016
- Listen, attend and spell: A neural network for large vocabulary conversational speech recognition2016
- Learning acoustic frame labeling for speech recognition with recurrent neural networks2015
- Effective Approaches to Attention-based Neural Machine Translation2015
- Connectionist temporal classification2006