Learning Simpler Language Models with the Differential State Framework
- 1 December 2017
- journal article
- research article
- Published by MIT Press in Neural Computation
- Vol. 29 (12) , 3327-3352
Abstract
Learning useful information across long time lags is a critical and difficult problem for temporal neural models in tasks such as language modeling. Existing architectures that address the issue are often complex and costly to train. The differential state framework (DSF) is a simple and high-performing design that unifies previously introduced gated neural models. DSF models maintain longer-term memory by learning to interpolate between a fast-changing data-driven representation and a slowly changing, implicitly stable state. Within the DSF framework, a new architecture is presented, the delta-RNN. This model requires hardly any more parameters than a classical, simple recurrent network. In language modeling at the word and character levels, the delta-RNN outperforms popular complex architectures, such as the long short-term memory (LSTM) and the gated recurrent unit (GRU), and, when regularized, performs comparably to several state-of-the-art baselines. At the subword level, the delta-RNN's performance is comparable to that of complex gated architectures.Keywords
All Related Versions
This publication has 19 references indexed in Scilit:
- Expectation-based syntactic comprehensionCognition, 2008
- Morphological Processing2006
- Second-order recurrent neural networks for grammatical inference2002
- A probabilistic earley parser as a psycholinguistic model2001
- Computation of Conditional Probability Statistics by 8-Month-Old InfantsPsychological Science, 1998
- The neural network pushdown automaton: Architecture, dynamics and training1998
- Long Short-Term MemoryNeural Computation, 1997
- First-order versus second-order single-layer recurrent neural networksIEEE Transactions on Neural Networks, 1994
- Acceleration of Stochastic Approximation by AveragingSIAM Journal on Control and Optimization, 1992
- Learning and Extracting Finite State Automata with Second-Order Recurrent Neural NetworksNeural Computation, 1992