FastAdaBelief: Improving Convergence Rate for Belief-Based Adaptive Optimizers by Exploiting Strong Convexity
- 10 March 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks and Learning Systems
- Vol. 34 (9) , 6515-6529
Abstract
AdaBelief, one of the current best optimizers, demonstrates superior generalization ability over the popular Adam algorithm by viewing the exponential moving average of observed gradients. AdaBelief is theoretically appealing in which it has a data-dependent O(√T) regret bound when objective functions are convex, where T is a time horizon. It remains, however, an open problem whether the convergence rate can be further improved without sacrificing its generalization ability. To this end, we make the first attempt in this work and design a novel optimization algorithm called FastAdaBelief that aims to exploit its strong convexity in order to achieve an even faster convergence rate. In particular, by adjusting the step size that better considers strong convexity and prevents fluctuation, our proposed FastAdaBelief demonstrates excellent generalization ability and superior convergence. As an important theoretical contribution, we prove that FastAdaBelief attains a data-dependent O(log T) regret bound, which is substantially lower than AdaBelief in strongly convex cases. On the empirical side, we validate our theoretical analysis with extensive experiments in scenarios of strong convexity and nonconvexity using three popular baseline models. Experimental results are very encouraging: FastAdaBelief converges the quickest in comparison to all mainstream algorithms while maintaining an excellent generalization ability, in cases of both strong convexity or nonconvexity. FastAdaBelief is, thus, posited as a new benchmark model for the research community.Keywords
All Related Versions
Funding Information
- Hundred Talents Program of the Chinese Academy of Sciences (Y9BEJ11001)
- Innovation Workstation of Suzhou Institute of Nano-Tech and Nano-Bionics (E010210101)
- National Natural Science Foundation of China (61876155)
- Jiangsu Science and Technology Program (BE2020006-4)
- U.K. Engineering and Physical Sciences Research Council (EP/M026981/1, EP/T021063/1, EP/T024917/1)
This publication has 20 references indexed in Scilit:
- Accelerating Sequential Minimal Optimization via Stochastic Subgradient DescentIEEE Transactions on Cybernetics, 2019
- Latent Factor-Based Recommenders Relying on Extended Stochastic Gradient Descent AlgorithmsIEEE Transactions on Systems, Man, and Cybernetics: Systems, 2019
- A Fast Non-Negative Latent Factor Model Based on Generalized Momentum MethodIEEE Transactions on Systems, Man, and Cybernetics: Systems, 2018
- Stochastic Conjugate Gradient Algorithm With Variance ReductionIEEE Transactions on Neural Networks and Learning Systems, 2018
- Seizure Classification From EEG Signals Using Transfer Learning, Semi-Supervised Learning and TSK Fuzzy SystemIEEE Transactions on Neural Systems and Rehabilitation Engineering, 2017
- Densely Connected Convolutional Networks2017
- Deep Residual Learning for Image Recognition2016
- Online Learning and Online Convex OptimizationFoundations and Trends in Machine Learning, 2012
- Hierarchical Bayesian Language Models for Conversational Speech RecognitionIEEE Transactions on Audio, Speech, and Language Processing, 2010
- Convex Optimization2004