Delving Deep into the Generalization of Vision Transformers under Distribution Shifts
Authors:
- 1 June 2022
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 7267-7276
Abstract
Vision Transformers (ViTs) have achieved impressive performance on various vision tasks, yet their generalization under distribution shifts (DS) is rarely understood. In this work, we comprehensively study the out-of-distribution (OOD) generalization of ViTs. For systematic investigation, we first present a taxonomy of DS. We then perform extensive evaluations of ViT variants under different DS and compare their generalization with Convolutional Neural Network (CNN) models. Important observations are obtained: 1) ViTs learn weaker biases on backgrounds and textures, while they are equipped with stronger inductive biases towards shapes and structures, which is more consistent with human cognitive traits. Therefore, ViTs generalize better than CNNs under DS. With the same or less amount of parameters, ViTs are ahead of corresponding CNNs by more than 5% in top-1 accuracy under most types of DS. 2) As the model scale increases, ViTs strengthen these biases and thus gradually narrow the in-distribution and OOD performance gap. To further improve the generalization of ViTs, we design the Generalization-Enhanced ViTs (GE-ViTs) from the perspectives of adversarial learning, information theory, and self-supervised learning. By comprehensively investigating these GE-ViTs and comparing with their corresponding CNN models, we observe: 1) For the enhanced model, larger ViTs still benefit more for the OOD generalization. 2) GE-ViTs are more sensitive to the hyper-parameters than their corresponding CNN models. We design a smoother learning strategy to achieve a stable training process and obtain performance improvements on OOD data by 4% from vanilla ViTs. We hope our comprehensive study could shed light on the design of more generalizable learning architectures. Codes and datasets are released in https://github.com/Phoenix1153/ViT_OOD_generalization.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization2021
- Emerging Properties in Self-Supervised Vision Transformers2021
- Prototypical Cross-domain Self-supervised Learning for Few-shot Unsupervised Domain Adaptation2021
- Big Transfer (BiT): General Visual Representation Learning2020
- Semi-Supervised Domain Adaptation via Minimax Entropy2019
- Moment Matching for Multi-Source Domain Adaptation2019
- Passive radar using Starlink transmissions: A theoretical study2019
- Domain Generalization with Adversarial Feature Learning2018
- Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization2017
- A Model of Inductive Bias LearningJournal of Artificial Intelligence Research, 2000