Exploring Masked Autoencoders for Sensor-Agnostic Image Retrieval in Remote Sensing
- 13 December 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Geoscience and Remote Sensing
- Vol. 63 (01962892) , 1-14
Abstract
Self-supervised learning through masked autoencoders (MAEs) has recently attracted great attention for remote sensing (RS) image representation learning, and thus embodies a significant potential for content-based image retrieval (CBIR) from ever-growing RS image archives. However, the existing MAE based CBIR studies in RS assume that the considered RS images are acquired by a single image sensor, and thus are only suitable for uni-modal CBIR problems. The effectiveness of MAEs for cross-sensor CBIR, which aims to search semantically similar images across different image modalities, has not been explored yet. In this paper, we take the first step to explore the effectiveness of MAEs for sensor-agnostic CBIR in RS. To this end, we present a systematic overview on the possible adaptations of the vanilla MAE to exploit masked image modeling on multi-sensor RS image archives (denoted as cross-sensor masked autoencoders [CSMAEs]) in the context of CBIR. Based on different adjustments applied to the vanilla MAE, we introduce different CSMAE models. We also provide an extensive experimental analysis of these CSMAE models. We finally derive a guideline to exploit masked image modeling for uni-modal and cross-modal CBIR problems in RS. The code of this work is publicly available at https://github.com/jakhac/CSMAE.Keywords
All Related Versions
Funding Information
- European Research Council through the (ERC-2017-STG)
- BigEarth Project (759764)
- European Space Agency (ESA) through the Demonstrator Precursor Digital Assistant Interface for Digital Twin Earth (DA4DTE)
This publication has 32 references indexed in Scilit:
- MMEarth: Exploring Multi-modal Pretext Tasks for Geospatial Representation Learning2024
- OmniSat: Self-supervised Modality Fusion for Earth Observation2024
- Bridging Remote Sensors with Multisensor Geospatial Foundation Models2024
- Scale-MAE: A Scale-Aware Masked Autoencoder for Multiscale Geospatial Representation Learning2023
- A Novel Self-Supervised Cross-Modal Image Retrieval Method in Remote Sensing2022
- Masked Autoencoders Are Scalable Vision Learners2022
- SimMIM: a Simple Framework for Masked Image Modeling2022
- Deep Learning for Image Search and Retrieval in Large Remote Sensing Archives2021
- CMIR-NET : A deep learning based model for cross-modal retrieval in remote sensingPattern Recognition Letters, 2020
- ImageNet: A large-scale hierarchical image database2009