Neural ODEs as the deep limit of ResNets with constant weights
Authors:
- 20 May 2020
- journal article
- research article
- Published by World Scientific Pub Co Pte Ltd in Analysis and Applications
- Vol. 19 (03) , 397-437
Abstract
In this paper, we prove that, in the deep limit, the stochastic gradient descent on a ResNet type deep neural network, where each layer shares the same weight matrix, converges to the stochastic gradient descent for a Neural ODE and that the corresponding value/loss functions converge. Our result gives, in the context of minimization by stochastic gradient descent, a theoretical foundation for considering Neural ODEs as the deep limit of ResNets. Our proof is based on certain decay estimates for associated Fokker–Planck equations.Keywords
All Related Versions
This publication has 20 references indexed in Scilit:
- Identity Mappings in Deep Residual Networks2016
- Deep Residual Learning for Image Recognition2016
- Stochastic First- and Zeroth-Order Methods for Nonconvex Stochastic ProgrammingSIAM Journal on Optimization, 2013
- On Uniqueness of a Probability Solution to the Cauchy Problem for the Fokker–Planck–Kolmogorov EquationTheory of Probability & Its Applications, 2012
- On uniqueness problems related to the Fokker–Planck–Kolmogorov equation for measuresJournal of Mathematical Sciences, 2011
- Partial Differential Equations2010
- Optimal Transport2009
- Existence and Uniqueness of Solutions to Fokker–Planck Type Equations with Irregular CoefficientsCommunications in Partial Differential Equations, 2008
- Topics in Optimal Transportation2003
- Logarithmic Sobolev InequalitiesAmerican Journal of Mathematics, 1975