Recurrent Neural Network based language modeling with controllable external Memory
Authors:
- 1 March 2017
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 5705-5709
Abstract
It is crucial for language models to model long-term dependency in word sequences, which can be achieved to some good extent by recurrent neural network (RNN) based language models with long short-term memory (LSTM) units. To accurately model the sophisticated long-term information in human languages, large memory in language models is necessary. However, the size of RNN-based language models cannot be arbitrarily increased because the computational resources required and the model complexity will also be increase accordingly, due to the limitation of the structure. To overcome this problem, inspired from Neural Turing Machine and Memory Network, we equip RNN-based language models with controllable external memory. With a learnable memory controller, the size of the external memory is independent to the number of model parameters, so the proposed language model can have larger memory without increasing the parameters. In the experiments, the proposed model yielded lower perplexities than RNN-based language models with LSTM units on both English and Chinese corpora.Keywords
This publication has 11 references indexed in Scilit:
- Recurrent Memory Networks for Language Modeling2016
- RNNDROP: A novel dropout for RNNS in ASR2015
- Improving the training and evaluation efficiency of recurrent neural network language models2015
- Scaling recurrent neural network language models2015
- Context dependent recurrent neural network language model2012
- Extensions of recurrent neural network language model2011
- Three new graphical models for statistical language modelling2007
- Exact training of a neural syntactic language model2004
- Improved backing-off for M-gram language modeling2002
- Long Short-Term MemoryNeural Computation, 1997