Towards Faster Training Algorithms Exploiting Bandit Sampling From Convex to Strongly Convex Conditions
- 23 May 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Emerging Topics in Computational Intelligence
- Vol. 7 (2) , 565-577
Abstract
The training process for deep learning and pattern recognition normally involves the use of convex and strongly convex optimization algorithms such as AdaBelief and SAdam to handle lots of “uninformative” samples that should be ignored, thus incurring extra calculations. To solve this open problem, we propose to design bandit sampling method to make these algorithms focus on “informative” samples during training process. Our contribution is twofold: first, we propose a convex optimization algorithm with bandit sampling, termed AdaBeliefBS, and prove that it converges faster than its original version; second, we prove that bandit sampling works well for strongly convex algorithms, and propose a generalized SAdam, called SAdamBS, that converges faster than SAdam. Finally, we conduct a series of experiments on various benchmark datasets to verify the fast convergence rate of our proposed algorithms.Keywords
Funding Information
- Suzhou Foreign Experts Project (E290010201)
- Chinese Academy of Sciences Key Project (E21Z010101)
- Innovation Workstation of Suzhou Institute of Nano-Tech and Nano-Bionics (E010210101)
- National Natural Science Foundation of China (61876155)
- Jiangsu Science and Technology Program (BE2020006-4)
- Engineering and Physical Sciences Research Council (EP/M026981/1, EP/T021063/1, EP/T024917/1)
This publication has 9 references indexed in Scilit:
- LightAdam: Towards a Fast and Accurate Adaptive Momentum Online AlgorithmCognitive Computation, 2022
- Drill the Cork of Information Bottleneck by Inputting the Most Important DataIEEE Transactions on Neural Networks and Learning Systems, 2021
- Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks2020
- Automatic Design of Deep Networks with Neural BlocksCognitive Computation, 2019
- Densely Connected Convolutional Networks2017
- Deep Residual Learning for Image Recognition2016
- On optimal probabilities in stochastic coordinate descent methodsOptimization Letters, 2015
- Finite-time Analysis of the Multiarmed Bandit ProblemMachine Learning, 2002
- Gradient-based learning applied to document recognitionProceedings of the IEEE, 1998