Regularizing Neural Networks via Adversarial Model Perturbation
Authors:
- 1 June 2021
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 8152-8161
Abstract
Effective regularization techniques are highly desired in deep learning for alleviating overfitting and improving generalization. This work proposes a new regularization scheme, based on the understanding that the flat local minima of the empirical risk cause the model to generalize better. This scheme is referred to as adversarial model perturbation (AMP), where instead of directly minimizing the empirical risk, an alternative "AMP loss" is minimized via SGD. Specifically, the AMP loss is obtained from the empirical risk by applying the "worst" norm-bounded perturbation on each point in the parameter space. Comparing with most existing regularization schemes, AMP has strong theoretical justifications, in that minimizing the AMP loss can be shown theoretically to favour flat local minima of the empirical risk. Extensive experiments on various modern deep architectures establish AMP as a new state of the art among regularization schemes. Our code is available at https://github.com/hiyouga/AMP-Regularizer.Keywords
All Related Versions
Funding Information
- National Key Research and Development Program of China
- National Natural Science Foundation of China
- State Key Laboratory of Software Development Environment
This publication has 12 references indexed in Scilit:
- Reconciling modern machine-learning practice and the classical bias–variance trade-offProceedings of the National Academy of Sciences, 2019
- MixUp as Locally Linear Out-of-Manifold RegularizationProceedings of the AAAI Conference on Artificial Intelligence, 2019
- Deep Pyramidal Residual Networks2017
- Rethinking the Inception Architecture for Computer Vision2016
- Wide Residual Networks2016
- DeepDriving: Learning Affordance for Direct Perception in Autonomous Driving2015
- Fast R-CNN2015
- Towards fully autonomous driving: Systems and algorithms2011
- Flat MinimaNeural Computation, 1997
- Modeling by shortest data descriptionAutomatica, 1978