Improving Vision Transformers by Revisiting High-frequency Components
Preprint
Abstract
The transformer models have shown promising effectiveness in dealing with various vision tasks. However, compared with training Convolutional Neural Network (CNN) models, training Vision Transformer (ViT) models is more difficult and relies on the large-scale training set. To explain this observation we make a hypothesis that \textit{ViT models are less effective in capturing the high-frequency components of images than CNN models}, and verify it by a frequency analysis. Inspired by this finding, we first investigate the effects of existing techniques for improving ViT models from a new frequency perspective, and find that the success of some techniques (e.g., RandAugment) can be attributed to the better usage of the high-frequency components. Then, to compensate for this insufficient ability of ViT models, we propose HAT, which directly augments high-frequency components of images via adversarial training. We show that HAT can consistently boost the performance of various ViT models (e.g., +1.2% for ViT-B, +0.5% for Swin-B), and especially enhance the advanced model VOLO-D5 to 87.3% that only uses ImageNet-1K data, and the superiority can also be maintained on out-of-distribution data and transferred to downstream tasks. The code is available at: https://github.com/jiawangbai/HAT.Keywords
All Related Versions
This publication has 49 references indexed in Scilit:
- ConViT: improving vision transformers with soft convolutional inductive biases*Journal of Statistical Mechanics: Theory and Experiment, 2022
- Learning Joint Spatial-Temporal Transformations for Video Inpainting2020
- AlphaFold at CASP13Bioinformatics, 2019
- Semantic Understanding of Scenes Through the ADE20K DatasetInternational Journal of Computer Vision, 2018
- CosFace: Large Margin Cosine Loss for Deep Face Recognition2018
- Deep Networks with Stochastic Depth2016
- A Discriminative Feature Learning Approach for Deep Face Recognition2016
- Deep Residual Learning for Image Recognition2016
- Microsoft COCO: Common Objects in Context2014
- A Demonstration of the Visual Importance and Flexibility of Spatial-Frequency Amplitude and PhasePerception, 1982