A New Perspective of Accelerated Gradient Methods: The Controlled Invariant Manifold Approach
Authors:
- Revati GunjalRGRevati Gunjal
- Sushama WaghSWSushama Wagh
- Syed Shadab NayyerSNSyed Shadab Nayyer
- Alex StankovicASAlex Stankovic
- Navdeep M. SinghNSNavdeep M. Singh
Preprint
Abstract
Gradient Descent (GD) is a ubiquitous algorithm for finding the optimal solution to an optimization problem. For reduced computational complexity, the optimal solution $\mathrm{x^*}$ of the optimization problem must be attained in a minimum number of iterations. For this objective, the paper proposes a genesis of an accelerated gradient algorithm through the controlled dynamical system perspective. The objective of optimally reaching the optimal solution $\mathrm{x^*}$ where $\mathrm{\nabla f(x^*)=0}$ with a given initial condition $\mathrm{x(0)}$ is achieved through control.
Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- Passivity and Immersion (P&I) Approach With Gaussian Process for Stabilization and Control of Nonlinear SystemsIEEE Access, 2022
- Passivity and Immersion (P&I) Approach for Constructive Stabilization and Control of Nonlinear SystemsIEEE Control Systems Letters, 2022
- Beyond convexity—Contraction and global convergence of gradient descentPLOS ONE, 2020
- Persistence of normally hyperbolic invariant manifolds in the absence of rate conditionsNonlinearity, 2020
- A variational perspective on accelerated methods in optimizationProceedings of the National Academy of Sciences, 2016
- Analysis and Design of Optimization Algorithms via Integral Quadratic ConstraintsSIAM Journal on Optimization, 2016
- A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse ProblemsSIAM Journal on Imaging Sciences, 2009
- THE HEAVY BALL WITH FRICTION METHOD, I. THE CONTINUOUS DYNAMICAL SYSTEM: GLOBAL EXPLORATION OF THE LOCAL MINIMA OF A REAL-VALUED FUNCTION BY ASYMPTOTIC ANALYSIS OF A DISSIPATIVE DYNAMICAL SYSTEMCommunications in Contemporary Mathematics, 2000
- Dynamical systems that sort lists, diagonalize matrices, and solve linear programming problemsLinear Algebra and its Applications, 1991