Extensions of recurrent neural network language model
Authors:
Top Cited Papers
- 1 May 2011
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- No. 15206149,p. 5528-5531
Abstract
We present several modifications of the original recurrent neural net work language model (RNN LM). While this model has been shown to significantly outperform many competitive language modeling techniques in terms of accuracy, the remaining problem is the computational complexity. In this work, we show approaches that lead to more than 15 times speedup for both training and testing phases. Next, we show importance of using a backpropagation through time algorithm. An empirical comparison with feedforward networks is also provided. In the end, we discuss possibilities how to reduce the amount of parameters in the model. The resulting RNN model can thus be smaller, faster both during training and testing, and more accurate than the basic one.Keywords
This publication has 13 references indexed in Scilit:
- Neural network based language models for highly inflective languages2009
- Adaptive Importance Sampling to Accelerate Training of a Neural Probabilistic Language ModelIEEE Transactions on Neural Networks, 2008
- Factored neural language models2006
- Training neural network language models on very large corpora2005
- Exact training of a neural syntactic language model2004
- Classes for fast maximum entropy training2002
- A bit of progress in language modelingComputer Speech & Language, 2001
- Learning long-term dependencies with gradient descent is difficultIEEE Transactions on Neural Networks, 1994
- Finding structure in timeCognitive Science, 1990
- Learning representations by back-propagating errorsNature, 1986