Stagewise Accelerated Stochastic Gradient Methods for Nonconvex Optimization
Open Access
- 27 May 2024
- journal article
- research article
- Published by MDPI AG in Mathematics
- Vol. 12 (11) , 1664
Abstract
For large-scale optimization that covers a wide range of optimization problems encountered frequently in machine learning and deep neural networks, stochastic optimization has become one of the most used methods thanks to its low computational complexity. In machine learning and deep learning problems, nonconvex problems are common, while convex problems are rare. How to find the global minimum for nonconvex optimization and reduce the computational complexity are challenges. Inspired by the phenomenon that the stagewise stepsize tuning strategy can empirically improve the convergence speed in deep neural networks, we incorporate the stagewise stepsize tuning strategy into the iterative framework of Nesterov’s acceleration- and variance reduction-based methods to reduce the computational complexity, i.e., the stagewise stepsize tuning strategy is incorporated into randomized stochastic accelerated gradient and stochastic variance-reduced gradient. The proposed methods are theoretically derived to reduce the complexity of the nonconvex and convex problems and improve the convergence rate of the frameworks, which have the complexity and , respectively, where is the PL modulus and L is the Lipschitz constant. In the end, numerical experiments on large benchmark datasets validate well the competitiveness of the proposed methods.
Keywords
Funding Information
- General Project of Zhejiang Provincial Department of Education (Y202147627)
This publication has 24 references indexed in Scilit:
- Lower bounds for non-convex stochastic optimizationMathematical Programming, 2022
- Robust recovery in 1-bit compressive sensing via ℓ -constrained least squaresSignal Processing, 2021
- To regularize or not: Revisiting SGD with simple algorithms and experimental studiesExpert Systems with Applications, 2018
- A “Nonconvex+Nonconvex” approach for image restoration with impulse noise removalApplied Mathematical Modelling, 2018
- From error bounds to the complexity of first-order descent methods for convex functionsMathematical Programming, 2016
- Accelerated gradient methods for nonconvex nonlinear and stochastic programmingMathematical Programming, 2015
- Primal-dual first-order methods with ${\mathcal {O}(1/\epsilon)}$ iteration-complexity for cone programmingMathematical Programming, 2009
- Primal-dual subgradient methods for convex problemsMathematical Programming, 2007
- Smooth minimization of non-smooth functionsMathematical Programming, 2004
- Some methods of speeding up the convergence of iteration methodsUSSR Computational Mathematics and Mathematical Physics, 1964