A Novel Self-Supervised Cross-Modal Image Retrieval Method in Remote Sensing
Authors:
- 16 October 2022
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 2426-2430
Abstract
Due to the availability of multi-modal remote sensing (RS) image archives, one of the most important research topics is the development of cross-modal RS image retrieval (CM-RSIR) methods that search semantically similar images across different modalities. Existing CM-RSIR methods require the availability of a high quality and quantity of annotated training images. The collection of a sufficient number of reliable labeled images is time consuming, complex and costly in operational scenarios, and can significantly affect the final accuracy of CM-RSIR. In this paper, we introduce a novel self-supervised CM-RSIR method that aims to: i) model mutual-information between different modalities in a self-supervised manner; ii) retain the distributions of modal-specific feature spaces similar to each other; and iii) define the most similar images within each modality without requiring any annotated training image. To this end, we propose a novel objective including three loss functions that simultaneously: i) maximize mutual information of different modalities for inter-modal similarity preservation; ii) minimize the angular distance of multi-modal image tuples for the elimination of inter-modal discrepancies; and iii) increase cosine similarity of the most similar images within each modality for the characterization of intra-modal similarities. Experimental results show the effectiveness of the proposed method compared to state-of-the-art methods. The code of the proposed method is publicly available at https://git.tu-berlin.de/rsim/SS-CM-RSIR.Keywords
All Related Versions
Funding Information
- European Research Council
This publication has 11 references indexed in Scilit:
- Multimodal Mutual Information Maximization: A Novel Approach for Unsupervised Deep Cross-Modal HashingIEEE Transactions on Neural Networks and Learning Systems, 2022
- Multisensor Fusion and Explicit Semantic Preserving-Based Deep Hashing for Cross-Modal Remote Sensing Image RetrievalIEEE Transactions on Geoscience and Remote Sensing, 2021
- BigEarthNet-MM: A Large-Scale, Multimodal, Multilabel Benchmark Archive for Remote Sensing Image Classification and Retrieval [Software and Data Sets]IEEE Geoscience and Remote Sensing Magazine, 2021
- Deep Learning for Image Search and Retrieval in Large Remote Sensing Archives2021
- Semantically Supervised Maximal Correlation For Cross-Modal Retrieval2020
- A Deep Cross-Modality Hashing Network for SAR and Optical Remote Sensing Images RetrievalIEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2020
- CMIR-NET : A deep learning based model for cross-modal retrieval in remote sensingPattern Recognition Letters, 2020
- Deep Supervised Cross-Modal Retrieval2019
- Densely Connected Convolutional Networks2017
- Cross-Modal Retrieval With CNN Visual Features: A New BaselineIEEE Transactions on Cybernetics, 2016