SALR: Sharpness-Aware Learning Rate Scheduler for Improved Generalization
- 7 April 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks and Learning Systems
- Vol. 35 (9) , 12518-12527
Abstract
In an effort to improve generalization in deep learning and automate the process of learning rate scheduling, we propose SALR: a sharpness-aware learning rate update technique designed to recover flat minimizers. Our method dynamically updates the learning rate of gradient-based optimizers based on the local sharpness of the loss function. This allows optimizers to automatically increase learning rates at sharp valleys to increase the chance of escaping them. We demonstrate the effectiveness of SALR when adopted by various algorithms over a broad range of networks. Our experiments indicate that SALR improves generalization, converges faster, and drives solutions to significantly flatter regions.Keywords
All Related Versions
Funding Information
- National Science Foundation (CMMI 1931950)
This publication has 21 references indexed in Scilit:
- Why ResNet Works? Residuals GeneralizeIEEE Transactions on Neural Networks and Learning Systems, 2020
- Entropy-SGD: biasing gradient descent into wide valleys*Journal of Statistical Mechanics: Theory and Experiment, 2019
- A survey on Image Data Augmentation for Deep LearningJournal of Big Data, 2019
- MobileNetV2: Inverted Residuals and Linear Bottlenecks2018
- Optimization Methods for Large-Scale Machine LearningSIAM Review, 2018
- Cyclical Learning Rates for Training Neural Networks2017
- Unreasonable effectiveness of learning neural networks: From accessible states and robust ensembles to basic algorithmic schemesProceedings of the National Academy of Sciences, 2016
- Subdominant Dense Clusters Allow for Simple Learning and High Computational Performance in Neural Networks with Discrete SynapsesPhysical Review Letters, 2015
- A Globally Convergent MC Algorithm With an Adaptive Learning RateIEEE Transactions on Neural Networks and Learning Systems, 2011
- On‐line learning for very large data setsApplied Stochastic Models in Business and Industry, 2005