Comparison of feedforward and recurrent neural network language models
Authors:
- 1 May 2013
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- No. 15206149,p. 8430-8434
Abstract
Research on language modeling for speech recognition has increasingly focused on the application of neural networks. Two competing concepts have been developed: On the one hand, feedforward neural networks representing an n-gram approach, on the other hand recurrent neural networks that may learn context dependencies spanning more than a fixed number of predecessor words. To the best of our knowledge, no comparison has been carried out between feedforward and state-of-the-art recurrent networks when applied to speech recognition. This paper analyzes this aspect in detail on a well-tuned French speech recognition task. In addition, we propose a simple and efficient method to normalize language model probabilities across different vocabularies, and we show how to speed up training of recurrent neural networks by parallelization.Keywords
This publication has 11 references indexed in Scilit:
- LSTM neural networks for language modeling2012
- Large vocabulary SOUL neural network language models2011
- Extensions of recurrent neural network language model2011
- Improved neural network based language modelling and adaptation2010
- iCNC and iROVER: the limits of improving system combination with classification?2008
- Continuous space language modelsComputer Speech & Language, 2007
- Improved backing-off for M-gram language modeling2002
- Classes for fast maximum entropy training2002
- Connectionist language modeling for large vocabulary continuous speech recognition2002
- A bit of progress in language modelingComputer Speech & Language, 2001