The Shattered Gradients Problem: If resnets are the answer, then what is the question?
Authors:
- David BalduzziDBDavid Balduzzi
- Marcus FreanMFMarcus Frean
- Lennox LearyLLLennox Leary
- Jp LewisJLJp Lewis
- Kurt Wan-Duo MaKMKurt Wan-Duo Ma
- Brian McWilliamsBMBrian McWilliams
Preprint
Abstract
A long-standing obstacle to progress in deep learning is the problem of vanishing and exploding gradients. The problem has largely been overcome through the introduction of carefully constructed initializations and batch normalization. Nevertheless, architectures incorporating skip-connections such as resnets perform much better than standard feedforward architectures despite well-chosen initialization and batch normalization. In this paper, we identify the shattered gradients problem. Specifically, we show that the correlation between gradients in standard feedforward networks decays exponentially with depth resulting in gradients that resemble white noise. In contrast, the gradients in architectures with skip-connections are far more resistant to shattering decaying sublinearly. Detailed empirical evidence is presented in support of the analysis, on both fully-connected networks and convnets. Finally, we present a new "looks linear" (LL) initialization that prevents shattering. Preliminary experiments show the new initialization allows to train very deep networks without the addition of skip-connections.Keywords
All Related Versions
This publication has 8 references indexed in Scilit:
- Inception-v4, Inception-ResNet and the Impact of Residual Connections on LearningProceedings of the AAAI Conference on Artificial Intelligence, 2017
- Identity Mappings in Deep Residual Networks2016
- Deep Residual Learning for Image Recognition2016
- Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification2015
- Going deeper with convolutions2015
- Greedy Layer-Wise Training of Deep Networks2007
- A Fast Learning Algorithm for Deep Belief NetsNeural Computation, 2006
- Learning long-term dependencies with gradient descent is difficultIEEE Transactions on Neural Networks, 1994