AdaCN: An Adaptive Cubic Newton Method for Nonconvex Stochastic Optimization
Open Access
- 1 January 2021
- journal article
- research article
- Published by Wiley in Computational Intelligence and Neuroscience
- Vol. 2021 (1)
Abstract
In this work, we introduce AdaCN, a novel adaptive cubic Newton method for nonconvex stochastic optimization. AdaCN dynamically captures the curvature of the loss landscape by diagonally approximated Hessian plus the norm of difference between previous two estimates. It only requires at most first order gradients and updates with linear complexity for both time and memory. In order to reduce the variance introduced by the stochastic nature of the problem, AdaCN hires the first and second moment to implement and exponential moving average on iteratively updated stochastic gradients and approximated stochastic Hessians, respectively. We validate AdaCN in extensive experiments, showing that it outperforms other stochastic first order methods (including SGD, Adam, and AdaBound) and stochastic quasi‐Newton method (i.e., Apollo), in terms of both convergence speed and generalization performance.Keywords
Funding Information
- National Natural Science Foundation of China (61602494)
This publication has 15 references indexed in Scilit:
- Stochastic Quasi-Newton Methods for Nonconvex Stochastic OptimizationSIAM Journal on Optimization, 2017
- Deep Residual Learning for Image Recognition2016
- Randomized algorithms for estimating the trace of an implicit symmetric positive semi-definite matrixJournal of the ACM, 2011
- Cubic regularization of Newton method and its global performanceMathematical Programming, 2006
- The Quasi-Cauchy Relation and Diagonal UpdatingSIAM Journal on Optimization, 1999
- Gradient-based learning applied to document recognitionProceedings of the IEEE, 1998
- Fast Exact Multiplication by the HessianNeural Computation, 1994
- Sizing and Least-Change Secant MethodsSIAM Journal on Numerical Analysis, 1993
- Some methods of speeding up the convergence of iteration methodsUSSR Computational Mathematics and Mathematical Physics, 1964
- A Stochastic Approximation MethodThe Annals of Mathematical Statistics, 1951