Efficient Training of Visual Transformers with Small Datasets
Preprint
Abstract
Visual Transformers (VTs) are emerging as an architectural paradigm alternative to Convolutional networks (CNNs). Differently from CNNs, VTs can capture global relations between image elements and they potentially have a larger representation capacity. However, the lack of the typical convolutional inductive bias makes these models more data-hungry than common CNNs. In fact, some local properties of the visual domain which are embedded in the CNN architectural design, in VTs should be learned from samples. In this paper, we empirically analyse different VTs, comparing their robustness in a small training-set regime, and we show that, despite having a comparable accuracy when trained on ImageNet, their performance on smaller datasets can be largely different. Moreover, we propose a self-supervised task which can extract additional information from images with only a negligible computational overhead. This task encourages the VTs to learn spatial relations within an image and makes the VT training much more robust when training data are scarce. Our task is used jointly with the standard (supervised) training and it does not depend on specific architectural choices, thus it can be easily plugged in the existing VTs. Using an extensive evaluation with different VTs and datasets, we show that our method can improve (sometimes dramatically) the final accuracy of the VTs. Our code is available at: https://github.com/yhlleo/VTs-Drloc.Keywords
All Related Versions
This publication has 37 references indexed in Scilit:
- Contrastive Multiview Coding2020
- SCAN: Learning to Classify Images Without Labels2020
- End-to-End Object Detection with Transformers2020
- CornerNet: Detecting Objects as Paired KeypointsInternational Journal of Computer Vision, 2019
- Semantic Understanding of Scenes Through the ADE20K DatasetInternational Journal of Computer Vision, 2018
- Deep Clustering for Unsupervised Learning of Visual Features2018
- DetNet: Design Backbone for Object Detection2018
- Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles2016
- Deep Residual Learning for Image Recognition2016
- Microsoft COCO: Common Objects in Context2014