TransMix: Attend to Mix for Vision Transformers
Authors:
- 1 June 2022
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 12125-12134
Abstract
Mixup-based augmentation has been found to be effective for generalizing models during training, especially for Vision Transformers (ViTs) since they can easily overfit. However, previous mixup-based methods have an underlying prior knowledge that the linearly interpolated ratio of targets should be kept the same as the ratio proposed in input interpolation. This may lead to a strange phenomenon that sometimes there is no valid object in the mixed image due to the random process in augmentation but there is still response in the label space. To bridge such gap between the input and label spaces, we propose TransMix, which mixes labels based on the attention maps of Vision Transformers. The confidence of the label will be larger if the corresponding input image is weighted higher by the attention map. TransMix is embarrassingly simple and can be implemented in just a few lines of code without introducing any extra parameters and FLOPs to ViT-based models. Experimental results show that our method can consistently improve various ViT-based models at scales on ImageNet classification. After pre-trained with TransMix on ImageNet, the ViT-based models also demonstrate better transferability to semantic segmentation, object detection and instance segmentation. TransMix also exhibits to be more robust when evaluating on 4 different benchmarks. Code is publicly available at https://github.com/Beckschen/TransMix.Keywords
All Related Versions
Funding Information
- ONR (N00014-21-1-2812)
- Lustgarten Foundation for Pancreatic Cancer Research (EP/W002981/1)
- ERC (ERC-2012-AdG,321162-HELIOS)
- EPSRC (EP/M013774/1,EP-SRC/MURI,EP/N019474/1)
This publication has 30 references indexed in Scilit:
- Attentive Cutmix: An Enhanced Data Augmentation Approach for Deep Learning Based Image Classification2020
- Rethinking ImageNet Pre-Training2019
- CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features2019
- Efficient Cipher-Only Attack of a Stream Cipher in RDHEI Based on Pixel Smoothness Estimation2019
- Mask R-CNN2017
- DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2017
- Deep Residual Learning for Image Recognition2016
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015
- The Role of Context for Object Detection and Semantic Segmentation in the Wild2014
- ImageNet: A large-scale hierarchical image database2009