Gradient Norm Aware Minimization Seeks First-Order Flatness and Improves Generalization
Authors:
- 1 June 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 20247-20257
Abstract
Recently, flat minima are proven to be effective for improving generalization and sharpness-aware minimization (SAM) achieves state-of-the-art performance. Yet the current definition of flatness discussed in SAM and its follow-ups are limited to the zeroth-order flatness (i.e., the worst-case loss within a perturbation radius). We show that the zeroth-order flatness can be insufficient to discriminate minima with low generalization error from those with high generalization error both when there is a single minimum or multiple minima within the given perturbation radius. Thus we present first-order flatness, a stronger measure of flatness focusing on the maximal gradient norm within a perturbation radius which bounds both the maximal eigenvalue of Hessian at local minima and the regularization function of SAM. We also present a novel training procedure named Gradient norm Aware Minimization (GAM) to seek minima with uniformly small curvature across all directions. Experimental results show that GAM improves the generalization of models trained with current optimizers such as SGD and Adam W on various datasets and networks. Furthermore, we show that GAM can help SAM find flatter minima and achieve better generalization. The code is available at https://github.com/xxgege/GAM.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (U1936219,62141607)
This publication has 18 references indexed in Scilit:
- Entropy-SGD: biasing gradient descent into wide valleys*Journal of Statistical Mechanics: Theory and Experiment, 2019
- CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features2019
- Recent Trends in Deep Learning Based Natural Language Processing [Review Article]IEEE Computational Intelligence Magazine, 2018
- Aggregated Residual Transformations for Deep Neural Networks2017
- Deep Residual Learning for Image Recognition2016
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015
- 3D Object Representations for Fine-Grained Categorization2013
- Cats and dogs2012
- Randomized algorithms for estimating the trace of an implicit symmetric positive semi-definite matrixJournal of the ACM, 2011
- Some large-scale matrix computation problemsJournal of Computational and Applied Mathematics, 1996