kTULA: A Langevin sampling algorithm with improved KL bounds under super-linear log-gradients
Preprint
Abstract
Motivated by applications in deep learning, where the global Lipschitz continuity condition is often not satisfied, we examine the problem of sampling from distributions with super-linearly growing log-gradients. We propose a novel tamed Langevin dynamics-based algorithm, called kTULA, to solve the aforementioned sampling problem, and provide a theoretical guarantee for its performance. More precisely, we establish a non-asymptotic convergence bound in Kullback-Leibler (KL) divergence with the best-known rate of convergence equal to $2-\overlineε$, $\overlineε>0$, which significantly improves relevant results in existing literature. This enables us to obtain an improved non-asymptotic error bound in Wasserstein-2 distance, which can be used to further derive a non-asymptotic guarantee for kTULA to solve the associated optimization problems. To illustrate the applicability of kTULA, we apply the proposed algorithm to the problem of sampling from a high-dimensional double-well potential distribution and to an optimization problem involving a neural network. We show that our main results can be used to provide theoretical guarantees for the performance of kTULA.
Keywords
All Related Versions
This publication has 19 references indexed in Scilit:
- Non-asymptotic estimates for accelerated high order Langevin Monte Carlo algorithmsAnalysis and Applications, 2025
- Kinetic Langevin MCMC sampling without gradient Lipschitz continuity - the strongly convex caseJournal of Complexity, 2024
- Non-asymptotic estimates for TUSLA algorithm for non-convex learning with applications to neural networks with ReLU activation functionIMA Journal of Numerical Analysis, 2023
- Taming Neural Networks with TUSLA: Nonconvex Learning via Adaptive Stochastic Gradient Langevin AlgorithmsSIAM Journal on Mathematics of Data Science, 2023
- On Stochastic Gradient Langevin Dynamics with Dependent Data Streams: The Fully Nonconvex CaseSIAM Journal on Mathematics of Data Science, 2021
- Wasserstein Continuity of Entropy and Outer Bounds for Interference ChannelsIEEE Transactions on Information Theory, 2016
- Strong convergence of an explicit numerical method for SDEs with nonglobally Lipschitz continuous coefficientsThe Annals of Applied Probability, 2012
- Information Theoretic Proofs of Entropy Power InequalitiesIEEE Transactions on Information Theory, 2010
- Strong and weak divergence in finite time of Euler's method for stochastic differential equations with non-globally Lipschitz continuous coefficientsProceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 2010
- A Simple Proof of the Existence of a Solution of Itô’s Equation with Monotone CoefficientsTheory of Probability & Its Applications, 1991