Spatial Entropy as an Inductive Bias for Vision Transformers
Authors:
Preprint
Abstract
Recent work on Vision Transformers (VTs) showed that introducing a local inductive bias in the VT architecture helps reducing the number of samples necessary for training. However, the architecture modifications lead to a loss of generality of the Transformer backbone, partially contradicting the push towards the development of uniform architectures, shared, e.g., by both the Computer Vision and the Natural Language Processing areas. In this work, we propose a different and complementary direction, in which a local bias is introduced using an auxiliary self-supervised task, performed jointly with standard supervised training. Specifically, we exploit the observation that the attention maps of VTs, when trained with self-supervision, can contain a semantic segmentation structure which does not spontaneously emerge when training is supervised. Thus, we explicitly encourage the emergence of this spatial clustering as a form of training regularization. In more detail, we exploit the assumption that, in a given image, objects usually correspond to few connected regions, and we propose a spatial formulation of the information entropy to quantify this object-based inductive bias. By minimizing the proposed spatial entropy, we include an additional self-supervised signal during training. Using extensive experiments, we show that the proposed regularization leads to equivalent or better results than other VT proposals which include a local bias by changing the basic Transformer architecture, and it can drastically boost the VT final accuracy when using small-medium training sets. The code is available at https://github.com/helia95/SAR.Keywords
All Related Versions
This publication has 44 references indexed in Scilit:
- Context Autoencoder for Self-supervised Representation LearningInternational Journal of Computer Vision, 2023
- Exploring Plain Vision Transformer Backbones for Object Detection2022
- What to Hide from Your Students: Attention-Guided Masked Image Modeling2022
- MultiMAE: Multi-modal Multi-task Masked Autoencoders2022
- PCT: Point cloud transformerComputational Visual Media, 2021
- Contrastive Multiview Coding2020
- SCAN: Learning to Classify Images Without Labels2020
- Deep Clustering for Unsupervised Learning of Visual Features2018
- The Pascal Visual Object Classes (VOC) ChallengeInternational Journal of Computer Vision, 2009
- Nonlinear total variation based noise removal algorithmsPhysica D: Nonlinear Phenomena, 1992