Learning Simpler Language Models with the Delta Recurrent Neural Network Framework
Authors:
Preprint
Abstract
Learning useful information across long time lags is a critical and difficult problem for temporal neural models in tasks like language modeling. Existing architectures that address the issue are often complex and costly to train. The Delta Recurrent Neural Network (Delta-RNN) framework is a simple and high-performing design that unifies previously proposed gated neural models. The Delta-RNN models maintain longer-term memory by learning to interpolate between a fast-changing data-driven representation and a slowly changing, implicitly stable state. This requires hardly any more parameters than a classical simple recurrent network. The models outperform popular complex architectures, such as the Long Short Term Memory (LSTM) and the Gated Recurrent Unit (GRU) and achieve state-of-the art performance in language modeling at character and word levels and yield comparable performance at the subword level.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- Hybrid computing using a neural network with dynamic external memoryNature, 2016
- Minimal gated unit for recurrent neural networksInternational Journal of Automation and Computing, 2016
- Parsing costs as predictors of reading difficulty: An evaluation using the Potsdam Sentence CorpusJournal of Eye Movement Research, 2008
- Expectation-based syntactic comprehensionCognition, 2008
- Noisy Time Series Prediction using Recurrent Neural Networks and Grammatical InferenceMachine Learning, 2001
- Computation of Conditional Probability Statistics by 8-Month-Old InfantsPsychological Science, 1998
- First-order versus second-order single-layer recurrent neural networksIEEE Transactions on Neural Networks, 1994
- Acceleration of Stochastic Approximation by AveragingSIAM Journal on Control and Optimization, 1992
- Learning and Extracting Finite State Automata with Second-Order Recurrent Neural NetworksNeural Computation, 1992