Taming Neural Networks with TUSLA: Nonconvex Learning via Adaptive Stochastic Gradient Langevin Algorithms
- 18 May 2023
- journal article
- research article
- Published by Society for Industrial & Applied Mathematics (SIAM) in SIAM Journal on Mathematics of Data Science
- Vol. 5 (2) , 323-345
Abstract
Artificial neural networks (ANNs) are typically highly nonlinear systems which are finely tuned via the optimization of their associated, nonconvex loss functions. In many cases, the gradient of any such loss function has superlinear growth, making the use of the widely accepted (stochastic) gradient descent methods, which are based on Euler numerical schemes, problematic. We offer a new learning algorithm based on an appropriately constructed variant of the popular stochastic gradient Langevin dynamics (SGLD), which is called the tamed unadjusted stochastic Langevin algorithm (TUSLA). We also provide a nonasymptotic analysis of the new algorithm’s convergence properties in the context of nonconvex learning problems with the use of ANNs. Thus, we provide finite-time guarantees for TUSLA to find approximate minimizers of both empirical and population risks. The roots of the TUSLA algorithm are based on the taming technology for diffusion processes with superlinear coefficients as developed in [S. Sabanis, Electron. Commun. Probab., 18 (2013), pp. 1–10] and [S. Sabanis, Ann. Appl. Probab., 26 (2016), pp. 2083–2105] and for Markov chain Monte Carlo algorithms in [N. Brosse, A. Durmus, É. Moulines, and S. Sabanis, Stochastic Process. Appl., 129 (2019), pp. 3638–3663]. Numerical experiments are presented which confirm the theoretical findings and illustrate the need for the use of the new algorithm in comparison to vanilla SGLD within the framework of ANNs.Keywords
All Related Versions
Funding Information
- Engineering and Physical Sciences Research Council (EP/N510129/1)
- Magyar Tudományos Akadémia (LP 2015-6)
- Nemzeti Kutatási Fejlesztési és Innovációs Hivatal (K 143529)
This publication has 14 references indexed in Scilit:
- User-friendly guarantees for the Langevin Monte Carlo with inaccurate gradientStochastic Processes and their Applications, 2019
- High-dimensional Bayesian inference via the unadjusted Langevin algorithmBernoulli, 2019
- The tamed unadjusted Langevin algorithmStochastic Processes and their Applications, 2019
- Couplings and quantitative contraction rates for Langevin dynamicsThe Annals of Probability, 2019
- Higher order Langevin Monte Carlo algorithmElectronic Journal of Statistics, 2019
- Quantitative Harris-type theorems for diffusions and McKean–Vlasov processesTransactions of the American Mathematical Society, 2018
- Euler approximations with varying coefficients: The case of superlinearly growing diffusion coefficientsThe Annals of Applied Probability, 2016
- Theoretical Guarantees for Approximate Sampling from Smooth and Log-Concave DensitiesJournal of the Royal Statistical Society Series B: Statistical Methodology, 2016
- A note on tamed Euler approximationsElectronic Communications in Probability, 2013
- Strong convergence of an explicit numerical method for SDEs with nonglobally Lipschitz continuous coefficientsThe Annals of Applied Probability, 2012