The Limiting Dynamics of SGD: Modified Loss, Phase-Space Oscillations, and Anomalous Diffusion
- 12 December 2023
- journal article
- research article
- Published by MIT Press in Neural Computation
- Vol. 36 (1) , 151-174
Abstract
In this work, we explore the limiting dynamics of deep neural networks trained with stochastic gradient descent (SGD). As observed previously, long after performance has converged, networks continue to move through parameter space by a process of anomalous diffusion in which distance traveled grows as a power law in the number of gradient updates with a nontrivial exponent. We reveal an intricate interaction among the hyperparameters of optimization, the structure in the gradient noise, and the Hessian matrix at the end of training that explains this anomalous diffusion. To build this understanding, we first derive a continuous-time model for SGD with finite learning rates and batch sizes as an underdamped Langevin equation. We study this equation in the setting of linear regression, where we can derive exact, analytic expressions for the phase-space dynamics of the parameters and their instantaneous velocities from initialization to stationarity. Using the Fokker-Planck equation, we show that the key ingredient driving these dynamics is not the original training loss but rather the combination of a modified loss, which implicitly regularizes the velocity, and probability currents that cause oscillations in phase space. We identify qualitative and quantitative predictions of this theory in the dynamics of a ResNet-18 model trained on ImageNet. Through the lens of statistical physics, we uncover a mechanistic origin for the anomalous limiting dynamics of deep neural networks trained with SGD. Understanding the limiting dynamics of SGD, and its dependence on various important hyperparameters like batch size, learning rate, and momentum, can serve as a basis for future work that can turn these insights into algorithmic gains.Keywords
All Related Versions
This publication has 7 references indexed in Scilit:
- Stochasticity helps to navigate rough landscapes: comparing gradient-descent-based algorithms in the phase retrieval problemMachine Learning: Science and Technology, 2021
- Entropy-SGD: biasing gradient descent into wide valleys*Journal of Statistical Mechanics: Theory and Experiment, 2019
- Stochastic Gradient Descent Performs Variational Inference, Converges to Limit Cycles for Deep Networks2018
- Structure of stochastic dynamics near fixed pointsProceedings of the National Academy of Sciences, 2005
- Potential in stochastic differential equations: novel constructionJournal of Physics A: General Physics, 2004
- On the momentum term in gradient descent learning algorithmsNeural Networks, 1999
- Bayesian Learning for Neural Networks1996