Exploring multidimensional lstms for large vocabulary ASR
Authors:
- 1 March 2016
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 4940-4944
Abstract
Long short-term memory (LSTM) recurrent neural networks (RNNs) have recently shown significant performance improvements over deep feed-forward neural networks. A key aspect of these models is the use of time recurrence, combined with a gating architecture that allows them to track the long-term dynamics of speech. Inspired by human spectrogram reading, we recently proposed the frequency LSTM (F-LSTM) that performs 1-D recurrence over the frequency axis and then performs 1-D recurrence over the time axis. In this study, we further improve the acoustic model by proposing a 2-D, time-frequency (TF) LSTM. The TF-LSTM jointly scans the input over the time and frequency axes to model spectro-temporal warping, and then uses the output activations as the input to a time LSTM (T-LSTM). The joint time-frequency modeling better normalizes the features for the upper layer T-LSTMs. Evaluated on a 375-hour short message dictation task, the proposed TF-LSTM obtained a 3.4% relative WER reduction over the best T-LSTM. The invariance property achieved by joint time-frequency analysis is demonstrated on a mismatched test set, where the TF-LSTM achieves a 14.2% relative WER reduction over the best T-LSTM.Keywords
This publication has 20 references indexed in Scilit:
- LSTM time and frequency recurrence for automatic speech recognition2015
- Constructing long short-term memory based deep recurrent neural networks for large vocabulary speech recognition2015
- Convolutional, Long Short-Term Memory, fully connected Deep Neural Networks2015
- An analysis of convolutional neural networks for speech recognition2015
- Hybrid speech recognition with Deep Bidirectional LSTM2013
- Recent advances in deep learning for speech research at Microsoft2013
- Understanding how Deep Belief Networks perform acoustic modelling2012
- Making Deep Belief Networks effective for large vocabulary continuous speech recognition2011
- Large vocabulary continuous speech recognition with context-dependent DBN-HMMS2011
- Acoustic Modeling Using Deep Belief NetworksIEEE Transactions on Audio, Speech, and Language Processing, 2011