Stochastic Gradient Variational Bayes for deep learning-based ASR
Authors:
- 1 December 2015
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- Vol. 12, 175-180
Abstract
Many successful methods for training deep neural networks (DNN) rely on an unsupervised pretraining algorithm. It is particularly effective when the number of labeled training samples is not large enough, because pretraining method helps to initialize the parameter values in the appropriate range near a local good minimum, for further discriminative finetuning. However, while the improvement is impressive, training DNN is difficult because the objective function of DNN is highly non-convex function of the parameters. To avoid placing the parameter that generalizes poorly, a robust generative modelling is necessary. This paper explore an alternative of generative modelling for pretraining DNN-based acoustic modelling using Stochastic Gradient Variational Bayes (SGVB) within autoencoder framework called Variational Bayes Autoencoder (VBAE). It performs an efficient approximate inference and learning with directed probabilistic graphical models. During fine-tuning, probabilistic encoder parameters with latent variable components are then used in discriminative training for acoustic model. Here, we investigate the performances of DNN-based acoustic model using the proposed pretrained VBAE in comparison with widely used pretraining algorithms like Restricted Boltzmann Machine (RBM) and Stacked Denoising Autoencoder (SDAE). The results reveal that VBAE pretraining with Gaussian latent variables gave the best performance.Keywords
This publication has 11 references indexed in Scilit:
- Bayesian Speech and Language Processing2015
- Recent advances in deep learning for speech research at Microsoft2013
- Representation Learning: A Review and New PerspectivesIEEE Transactions on Pattern Analysis and Machine Intelligence, 2013
- Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research GroupsIEEE Signal Processing Magazine, 2012
- Understanding how Deep Belief Networks perform acoustic modelling2012
- A Fast Learning Algorithm for Deep Belief NetsNeural Computation, 2006
- Variational Bayesian Estimation and Clustering for Speech RecognitionIEEE Transactions on Speech and Audio Processing, 2004
- Robust text-independent speaker identification using Gaussian mixture speaker modelsIEEE Transactions on Speech and Audio Processing, 1995
- A tutorial on hidden Markov models and selected applications in speech recognitionProceedings of the IEEE, 1989
- Modeling by shortest data descriptionAutomatica, 1978