Rethinking the Effect of Data Augmentation in Adversarial Contrastive Learning
Authors:
Preprint
Abstract
Recent works have shown that self-supervised learning can achieve remarkable robustness when integrated with adversarial training (AT). However, the robustness gap between supervised AT (sup-AT) and self-supervised AT (self-AT) remains significant. Motivated by this observation, we revisit existing self-AT methods and discover an inherent dilemma that affects self-AT robustness: either strong or weak data augmentations are harmful to self-AT, and a medium strength is insufficient to bridge the gap. To resolve this dilemma, we propose a simple remedy named DYNACL (Dynamic Adversarial Contrastive Learning). In particular, we propose an augmentation schedule that gradually anneals from a strong augmentation to a weak one to benefit from both extreme cases. Besides, we adopt a fast post-processing stage for adapting it to downstream tasks. Through extensive experiments, we show that DYNACL can improve state-of-the-art self-AT robustness by 8.84% under Auto-Attack on the CIFAR-10 dataset, and can even outperform vanilla supervised adversarial training for the first time. Our code is available at \url{https://github.com/PKU-ML/DYNACL}.Keywords
All Related Versions
This publication has 7 references indexed in Scilit:
- Decoupled Adversarial Contrastive Learning for Self-supervised Adversarial Robustness2022
- An Empirical Study of Training Self-Supervised Vision Transformers2021
- Contrastive Multiview Coding2020
- Momentum Contrast for Unsupervised Visual Representation Learning2020
- Adversarial Examples Improve Image Recognition2020
- The Unreasonable Effectiveness of Deep Features as a Perceptual Metric2018
- Deep Residual Learning for Image Recognition2016