Attention-Guided Masked Autoencoders For Learning Image Representations
Preprint
Abstract
Masked autoencoders (MAEs) have established themselves as a powerful method for unsupervised pre-training for computer vision tasks. While vanilla MAEs put equal emphasis on reconstructing the individual parts of the image, we propose to inform the reconstruction process through an attention-guided loss function. By leveraging advances in unsupervised object discovery, we obtain an attention map of the scene which we employ in the loss function to put increased emphasis on reconstructing relevant objects, thus effectively incentivizing the model to learn more object-focused representations without compromising the established masking strategy. Our evaluations show that our pre-trained models learn better latent representations than the vanilla MAE, demonstrated by improved linear probing and k-NN classification results on several benchmarks while at the same time making ViTs more robust against varying backgrounds.Keywords
All Related Versions
This publication has 8 references indexed in Scilit:
- TokenCut: Segmenting Objects in Images and Videos With Self-Supervised Transformer and Normalized CutIEEE Transactions on Pattern Analysis and Machine Intelligence, 2023
- OmniMAE: Single Model Masked Pretraining on Images and Videos2023
- Cut and Learn for Unsupervised Object Detection and Instance Segmentation2023
- Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture2023
- What to Hide from Your Students: Attention-Guided Masked Image Modeling2022
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding2019
- 3D Object Representations for Fine-Grained Categorization2013
- Normalized cuts and image segmentationIEEE Transactions on Pattern Analysis and Machine Intelligence, 2000