Taming neural networks with TUSLA: Non-convex learning via adaptive stochastic gradient Langevin algorithms
Authors:
Preprint
Abstract
Artificial neural networks (ANNs) are typically highly nonlinear systems which are finely tuned via the optimization of their associated, non-convex loss functions. In many cases, the gradient of any such loss function has superlinear growth, making the use of the widely-accepted (stochastic) gradient descent methods, which are based on Euler numerical schemes, problematic. We offer a new learning algorithm based on an appropriately constructed variant of the popular stochastic gradient Langevin dynamics (SGLD), which is called tamed unadjusted stochastic Langevin algorithm (TUSLA). We also provide a nonasymptotic analysis of the new algorithm's convergence properties in the context of non-convex learning problems with the use of ANNs. Thus, we provide finite-time guarantees for TUSLA to find approximate minimizers of both empirical and population risks. The roots of the TUSLA algorithm are based on the taming technology for diffusion processes with superlinear coefficients as developed in \citet{tamed-euler, SabanisAoAP} and for MCMC algorithms in \citet{tula}. Numerical experiments are presented which confirm the theoretical findings and illustrate the need for the use of the new algorithm in comparison to vanilla SGLD within the framework of ANNs.Keywords
All Related Versions
This publication has 14 references indexed in Scilit:
- Nonasymptotic Estimates for Stochastic Gradient Langevin Dynamics Under Local Conditions in Nonconvex OptimizationApplied Mathematics & Optimization, 2023
- On Stochastic Gradient Langevin Dynamics with Dependent Data Streams: The Fully Nonconvex CaseSIAM Journal on Mathematics of Data Science, 2021
- Couplings and quantitative contraction rates for Langevin dynamicsThe Annals of Probability, 2019
- Higher order Langevin Monte Carlo algorithmElectronic Journal of Statistics, 2019
- Quantitative Harris-type theorems for diffusions and McKean–Vlasov processesTransactions of the American Mathematical Society, 2018
- Nonasymptotic convergence analysis for the unadjusted Langevin algorithmThe Annals of Applied Probability, 2017
- Euler approximations with varying coefficients: The case of superlinearly growing diffusion coefficientsThe Annals of Applied Probability, 2016
- Theoretical Guarantees for Approximate Sampling from Smooth and Log-Concave DensitiesJournal of the Royal Statistical Society Series B: Statistical Methodology, 2016
- Strong convergence of an explicit numerical method for SDEs with nonglobally Lipschitz continuous coefficientsThe Annals of Applied Probability, 2012
- Strong and weak divergence in finite time of Euler's method for stochastic differential equations with non-globally Lipschitz continuous coefficientsProceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 2010