The Deep Tensor Neural Network With Applications to Large Vocabulary Speech Recognition
Authors:
- 16 November 2012
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Audio, Speech, and Language Processing
- Vol. 21 (2) , 388-396
Abstract
The recently proposed context-dependent deep neural network hidden Markov models (CD-DNN-HMMs) have been proved highly promising for large vocabulary speech recognition. In this paper, we develop a more advanced type of DNN, which we call the deep tensor neural network (DTNN). The DTNN extends the conventional DNN by replacing one or more of its layers with a double-projection (DP) layer, in which each input vector is projected into two nonlinear subspaces, and a tensor layer, in which two subspace projections interact with each other and jointly predict the next layer in the deep architecture. In addition, we describe an approach to map the tensor layers to the conventional sigmoid layers so that the former can be treated and trained in a similar way to the latter. With this mapping we can consider a DTNN as the DNN augmented with DP layers so that not only the BP learning algorithm of DTNNs can be cleanly derived but also new types of DTNNs can be more easily developed. Evaluation on Switchboard tasks indicates that DTNNs can outperform the already high-performing DNNs with 4-5% and 3% relative word error reduction, respectively, using 30-hr and 309-hr training sets.Keywords
This publication has 12 references indexed in Scilit:
- A deep architecture with bilinear modeling of hidden representations: Applications to phonetic recognition2012
- Exploiting sparseness in deep neural networks for large vocabulary speech recognition2012
- Feature engineering in Context-Dependent Deep Neural Networks for conversational speech transcription2011
- Making Deep Belief Networks effective for large vocabulary continuous speech recognition2011
- Context-Dependent Pre-Trained Deep Neural Networks for Large-Vocabulary Speech RecognitionIEEE Transactions on Audio, Speech, and Language Processing, 2011
- Acoustic Modeling Using Deep Belief NetworksIEEE Transactions on Audio, Speech, and Language Processing, 2011
- Modeling pixel means and covariances using factorized third-order boltzmann machines2010
- Tensor Decompositions and ApplicationsSIAM Review, 2009
- Connectionist probability estimators in HMM speech recognitionIEEE Transactions on Speech and Audio Processing, 1994
- Tensor network theory of the metaorganization of functional geometries in the central nervous systemNeuroscience, 1985