Gated Recurrent Neural Networks with Weighted Time-Delay Feedback
Preprint
Abstract
We introduce a novel gated recurrent unit (GRU) with a weighted time-delay feedback mechanism in order to improve the modeling of long-term dependencies in sequential data. This model is a discretized version of a continuous-time formulation of a recurrent unit, where the dynamics are governed by delay differential equations (DDEs). By considering a suitable time-discretization scheme, we propose $\tau$-GRU, a discrete-time gated recurrent unit with delay. We prove the existence and uniqueness of solutions for the continuous-time model, and we demonstrate that the proposed feedback mechanism can help improve the modeling of long-term dependencies. Our empirical results show that $\tau$-GRU can converge faster and generalize better than state-of-the-art recurrent units and gated recurrent architectures on a range of tasks, including time-series classification, human activity recognition, and speech recognition.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- Climate models with delay differential equationsChaos: An Interdisciplinary Journal of Nonlinear Science, 2017
- Exponential input-to-state stability of recurrent neural networks with multiple time-varying delaysCognitive Neurodynamics, 2013
- Optimization and applications of echo state networks with leaky- integrator neuronsNeural Networks, 2007
- Delay-dependent exponential stability analysis of delayed neural networks: an LMI approachNeural Networks, 2002
- Time-delay recurrent neural network for temporal correlations and predictionNeurocomputing, 1998
- Gradient-based learning applied to document recognitionProceedings of the IEEE, 1998
- Learning long-term dependencies in NARX recurrent neural networksIEEE Transactions on Neural Networks, 1996
- How delays affect neural dynamics and learningIEEE Transactions on Neural Networks, 1994
- A time-delay neural network architecture for isolated word recognitionNeural Networks, 1990
- Phoneme recognition using time-delay neural networksIEEE Transactions on Acoustics, Speech, and Signal Processing, 1989