Fast Adversarial Training With Adaptive Step Size
- 26 October 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 32 (10577149) , 6102-6114
Abstract
While adversarial training and its variants have shown to be the most effective algorithms to defend against adversarial attacks, their extremely slow training process makes it hard to scale to large datasets like ImageNet. The key idea of recent works to accelerate adversarial training is to substitute multi-step attacks (e.g., PGD) with single-step attacks (e.g., FGSM). However, these single-step methods suffer from catastrophic overfitting, where the accuracy against PGD attack suddenly drops to nearly 0% during training, and the network totally loses its robustness. In this work, we study the phenomenon from the perspective of training instances. We show that catastrophic overfitting is instance-dependent, and fitting instances with larger input gradient norm is more likely to cause catastrophic overfitting. Based on our findings, we propose a simple but effective method, Adversarial Training with Adaptive Step size (ATAS) . ATAS learns an instance-wise adaptive step size that is inversely proportional to its gradient norm. Our theoretical analysis shows that ATAS converges faster than the commonly adopted non-adaptive counterparts. Empirically, ATAS consistently mitigates catastrophic overfitting and achieves higher robust accuracy on CIFAR10, CIFAR100, and ImageNet when evaluated on various adversarial budgets.Keywords
All Related Versions
Funding Information
- CityU APRC (9610614)
This publication has 17 references indexed in Scilit:
- Defense Against Adversarial Attacks by Reconstructing ImagesIEEE Transactions on Image Processing, 2021
- Training Robust Deep Neural Networks via Adversarial Noise PropagationIEEE Transactions on Image Processing, 2021
- Square Attack: A Query-Efficient Black-Box Adversarial Attack via Random Search2020
- Efficient Adversarial Training With Transferable Adversarial Examples2020
- Image Super-Resolution as a Defense Against Adversarial AttacksIEEE Transactions on Image Processing, 2019
- Defense Against Adversarial Attacks Using High-Level Representation Guided Denoiser2018
- Wide Residual Networks2016
- Algorithmic game theoryCommunications of the ACM, 2010
- ImageNet: A large-scale hierarchical image database2009
- Dynamic Noncooperative Game Theory, 2nd Edition1998