Overcoming the vanishing gradient problem in plain recurrent networks
Preprint
Abstract
Plain recurrent networks greatly suffer from the vanishing gradient problem while Gated Neural Networks (GNNs) such as Long-short Term Memory (LSTM) and Gated Recurrent Unit (GRU) deliver promising results in many sequence learning tasks through sophisticated network designs. This paper shows how we can address this problem in a plain recurrent network by analyzing the gating mechanisms in GNNs. We propose a novel network called the Recurrent Identity Network (RIN) which allows a plain recurrent network to overcome the vanishing gradient problem while training very deep models without the use of gates. We compare this model with IRNNs and LSTMs on multiple sequence modeling benchmarks. The RINs demonstrate competitive performance and converge faster in all tasks. Notably, small RIN models produce 12%--67% higher accuracy on the Sequential and Permuted MNIST datasets and reach state-of-the-art performance on the bAbI question answering dataset.Keywords
All Related Versions
This publication has 8 references indexed in Scilit:
- Gated Orthogonal Recurrent Units: On Learning to ForgetNeural Computation, 2019
- All You Need is Beyond a Good Init: Exploring Better Solution for Training Extremely Deep Convolutional Neural Networks with Orthonormality and Modulation2017
- Identity Mappings in Deep Residual Networks2016
- Deep Residual Learning for Image Recognition2016
- Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation2014
- A Fast Learning Algorithm for Deep Belief NetsNeural Computation, 2006
- Gradient-based learning applied to document recognitionProceedings of the IEEE, 1998
- Learning Complex, Extended Sequences Using the Principle of History CompressionNeural Computation, 1992