Knowledge distillation for small-footprint highway networks
Authors:
- 1 March 2017
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 4820-4824
Abstract
Deep learning has significantly advanced state-of-the-art of speech recognition in the past few years. However, compared to conventional Gaussian mixture acoustic models, neural network models are usually much larger, and are therefore not very deployable in embedded devices. Previously, we investigated a compact highway deep neural network (HDNN) for acoustic modelling, which is a type of depth-gated feedforward neural network. We have shown that HDNN-based acoustic models can achieve comparable recognition accuracy with much smaller number of model parameters compared to plain deep neural network (DNN) acoustic models. In this paper, we push the boundary further by leveraging on the knowledge distillation technique that is also known as teacher-student training, i.e., we train the compact HDNN model with the supervision of a high accuracy cumbersome model. Furthermore, we also investigate sequence training and adaptation in the context of teacher-student training. Our experiments were performed on the AMI meeting speech recognition corpus. With this technique, we significantly improved the recognition accuracy of the HDNN acoustic model with less than 0.8 million parameters, and narrowed the gap between this model and the plain DNN with 30 million parameters.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- Sequence training and adaptation of highway deep neural networks2016
- Small-Footprint Deep Neural Networks with Highway Connections for Speech Recognition2016
- Convolutional Neural Networks for Speech RecognitionIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2014
- Asynchronous stochastic optimization for sequence training of deep neural networks2014
- Error back propagation for sequence training of Context-Dependent Deep NetworkS for conversational speech transcription2013
- Low-rank matrix factorization for Deep Neural Network training with high-dimensional output targets2013
- Lattice-based optimization of sequence classification criteria for neural-network acoustic modeling2009
- Recognition and understanding of meetings the AMI and AMIDA projects2007
- Model compression2006