Neural Optimizer Equation, Decay Function, and Learning Rate Schedule Joint Evolution
Open Access
- 14 July 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1100-1109
Abstract
A major contributor to the quality of a deep learning model is the selection of the optimizer. We propose a new dual-joint search space in the realm of neural optimizer search (NOS), along with an integrity check, to automate the process of finding deep learning optimizers. Our dual-joint search space simultaneously allows for the optimization of not only the update equation, but also internal decay functions and learning rate schedules for optimizers. We search the space using our proposed mutation-only, particle-based genetic algorithm able to be massively parallelized for our domain-specific problem. We evaluate our candidate optimizers on the CIFAR-10 dataset using a small ConvNet. To assess generalization, the final optimizers were then transferred to large-scale image classification on CIFAR-100 and TinyImageNet, while also being fine-tuned on Flowers102, Cars196, and Caltech101 using EfficientNetV2Small. We found multiple optimizers, learning rate schedules, and Adam variants that outperformed Adam, as well as other standard deep learning optimizers, across the image classification tasks.Keywords
All Related Versions
This publication has 15 references indexed in Scilit:
- Demon: Improved Neural Network Training With Momentum Decay2022
- AutoML: A survey of the state-of-the-artKnowledge-Based Systems, 2021
- Stochastic Normalized Gradient Descent with Momentum for Large Batch TrainingarXiv, 2020
- Evolutionary optimization of deep learning activation functions2020
- Randaugment: Practical automated data augmentation with a reduced search space2020
- Regularized Evolution for Image Classifier Architecture SearchProceedings of the AAAI Conference on Artificial Intelligence, 2019
- Accelerating Stochastic Gradient Descent For Least Squares RegressionarXiv, 2018
- YellowFin and the Art of Momentum TuningarXiv, 2018
- 3D Object Representations for Fine-Grained Categorization2013
- Learning Generative Visual Models from Few Training Examples: An Incremental Bayesian Approach Tested on 101 Object Categories2005