HyAdamC: A New Adam-Based Hybrid Optimization Algorithm for Convolution Neural Networks
Open Access
Abstract
As the performance of devices that conduct large-scale computations has been rapidly improved, various deep learning models have been successfully utilized in various applications. Particularly, convolution neural networks (CNN) have shown remarkable performance in image processing tasks such as image classification and segmentation. Accordingly, more stable and robust optimization methods are required to effectively train them. However, the traditional optimizers used in deep learning still have unsatisfactory training performance for the models with many layers and weights. Accordingly, in this paper, we propose a new Adam-based hybrid optimization method called HyAdamC for training CNNs effectively. HyAdamC uses three new velocity control functions to adjust its search strength carefully in term of initial, short, and long-term velocities. Moreover, HyAdamC utilizes an adaptive coefficient computation method to prevent that a search direction determined by the first momentum is distorted by any outlier gradients. Then, these are combined into one hybrid method. In our experiments, HyAdamC showed not only notable test accuracies but also significantly stable and robust optimization abilities when training various CNN models. Furthermore, we also found that HyAdamC could be applied into not only image classification and image segmentation tasks.Keywords
Funding Information
- National Research Foundation of Korea (2018R1A5A7059549, 2020R1A2C1014037)
- Ministry of Trade, Industry and Energy (10077553)
This publication has 40 references indexed in Scilit:
- Comprehensive Learning Particle Swarm Optimization Algorithm With Local Search for Multimodal FunctionsIEEE Transactions on Evolutionary Computation, 2018
- MobileNetV2: Inverted Residuals and Linear Bottlenecks2018
- Deep Convolutional Neural Networks for Image Classification: A Comprehensive ReviewNeural Computation, 2017
- Densely Connected Convolutional Networks2017
- An Integrated Micro- and Macroarchitectural Analysis of the Drosophila Brain by Computer-Assisted Serial Section Electron MicroscopyPLOS Biology, 2010
- Conjugate gradient methodWIREs Computational Statistics, 2009
- A direct adaptive method for faster backpropagation learning: the RPROP algorithm2002
- On the momentum term in gradient descent learning algorithmsNeural Networks, 1999
- On the limited memory BFGS method for large scale optimizationMathematical Programming, 1989
- A Broyden—Fletcher—Goldfarb—Shanno optimization procedure for molecular geometriesChemical Physics Letters, 1985