FastAdaBelief: Improving Convergence Rate for Belief-based Adaptive Optimizers by Exploiting Strong Convexity
Preprint
Abstract
AdaBelief, one of the current best optimizers, demonstrates superior generalization ability compared to the popular Adam algorithm by viewing the exponential moving average of observed gradients. AdaBelief is theoretically appealing in that it has a data-dependent $O(\sqrt{T})$ regret bound when objective functions are convex, where $T$ is a time horizon. It remains however an open problem whether the convergence rate can be further improved without sacrificing its generalization ability. %on how to exploit strong convexity to further improve the convergence rate of AdaBelief. To this end, we make a first attempt in this work and design a novel optimization algorithm called FastAdaBelief that aims to exploit its strong convexity in order to achieve an even faster convergence rate. In particular, by adjusting the step size that better considers strong convexity and prevents fluctuation, our proposed FastAdaBelief demonstrates excellent generalization ability as well as superior convergence. As an important theoretical contribution, we prove that FastAdaBelief attains a data-dependant $O(\log T)$ regret bound, which is substantially lower than AdaBelief. On the empirical side, we validate our theoretical analysis with extensive experiments in both scenarios of strong and non-strong convexity on three popular baseline models. Experimental results are very encouraging: FastAdaBelief converges the quickest in comparison to all mainstream algorithms while maintaining an excellent generalization ability, in cases of both strong or non-strong convexity. FastAdaBelief is thus posited as a new benchmark model for the research community.
Keywords
All Related Versions
This publication has 14 references indexed in Scilit:
- Defect Image Sample Generation With GAN for Improving Defect RecognitionIEEE Transactions on Automation Science and Engineering, 2020
- A Randomized Block-Coordinate Adam online learning optimization algorithmNeural Computing and Applications, 2020
- Stochastic Gradient Descent for Nonconvex Learning Without Bounded Gradient AssumptionsIEEE Transactions on Neural Networks and Learning Systems, 2019
- Nonzero-Sum Game Reinforcement Learning for Performance Optimization in Large-Scale Industrial ProcessesIEEE Transactions on Cybernetics, 2019
- Label Distribution Learning with Label Correlations on Local SamplesIEEE Transactions on Knowledge and Data Engineering, 2019
- Accelerating Sequential Minimal Optimization via Stochastic Subgradient DescentIEEE Transactions on Cybernetics, 2019
- Stochastic Conjugate Gradient Algorithm With Variance ReductionIEEE Transactions on Neural Networks and Learning Systems, 2018
- An online prediction algorithm for reinforcement learning with linear function approximation using cross entropy methodMachine Learning, 2018
- Deep Residual Learning for Image Recognition2016
- Hierarchical Bayesian Language Models for Conversational Speech RecognitionIEEE Transactions on Audio, Speech, and Language Processing, 2010