Exploring the Relationship Between Architectural Design and Adversarially Robust Generalization
Authors:
- 1 June 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 4096-4107
Abstract
Adversarial training has been demonstrated to be one of the most effective remedies for defending adversarial examples, yet it often suffers from the huge robustness generalization gap on unseen testing adversaries, deemed as the adversarially robust generalization problem. Despite the preliminary understandings devoted to adversarially robust generalization, little is known from the architectural perspective. To bridge the gap, this paper for the first time systematically investigated the relationship between adversarially robust generalization and architectural design. In particular, we comprehensively evaluated 20 most representative adversarially trained architectures on ImageNette and CIFAR-10 datasets towards multiple $\ell_{p}$ -norm adversarial attacks. Based on the extensive experiments, we found that, under aligned settings, Vision Transformers (e.g., PVT, CoAtNet) often yield better adversarially robust generalization while CNNs tend to overfit on specific attacks and fail to generalize on multiple adversaries. To better understand the nature behind it, we conduct theoretical analysis via the lens of Rademacher complexity. We revealed the fact that the higher weight sparsity contributes significantly towards the better adversarially robust generalization of Transformers, which can be often achieved by the specially-designed attention blocks. We hope our paper could help to better understand the mechanism for designing robust DNNs. Our model weights can be found at http://robust.art.
Keywords
All Related Versions
Funding Information
- National Key Research and Development Plan of China (2021ZD0110601)
- National Natural Science Foundation of China (62022009,62206009)
- State Key Laboratory of Software Development Environment
This publication has 12 references indexed in Scilit:
- Boosting the Transferability of Video Adversarial Examples via Temporal TranslationProceedings of the AAAI Conference on Artificial Intelligence, 2022
- Swin Transformer: Hierarchical Vision Transformer using Shifted Windows2021
- Incorporating Convolution Designs into Visual Transformers2021
- Robustness and Generalization via Generative Adversarial Training2021
- Bottleneck Transformers for Visual Recognition2021
- Perceptual-Sensitive GAN for Generating Adversarial PatchesProceedings of the AAAI Conference on Artificial Intelligence, 2019
- Disentangling Adversarial Robustness and Generalization2019
- Adversarial Examples in the Physical World2018
- Densely Connected Convolutional Networks2017
- Deep Residual Learning for Image Recognition2016