Multimodal Fusion Remote Sensing Image–Audio Retrieval
Open Access
- 2 August 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing
- Vol. 15 (19391404) , 6220-6235
Abstract
Remote sensing image–audio retrieval (RSIAR) has been an emerging research topic in recent years, and many different methods have been proposed for this topic. These RSIAR methods have achieved good retrieval results, but two problems remain: the lack of discriminability of audio modality and the existence of a heterogeneous gap between audio and image. These two problems make the cross-modal common embedding space for audio and images suboptimal, often failing to perform superior retrieval. This article proposes a novel RSIAR method named multimodal fusion remote sensing image–audio retrieval (MMFR) to address these two problems. MMFR first converts original audio input to text. Then, MMFR uses a feature fusion module to obtain a fusion representation fused with text information instead of the original sole audio representation. Fusion text information can make the pronunciation-based audio feature more semantically discriminable and convert pronunciation-based audio feature to more “high-level” fusion feature to cross the heterogeneous gap. Seven different fusion methods are tried in the feature fusion module. In addition, the triplet loss, the semantic loss, and the consistency loss are used to optimize the common retrieval space. Extensive experiments conducted on the UCM_IV, RSICD_IV, and SYDNE_IV datasets demonstrate that our MMFR method outperforms state-of-the-art methods.Keywords
Funding Information
- Key Research and Development Program of Shaanxi (2021ZDLGY01-06, 2022ZDLGY01-12)
- National Key Research and Development Program of China (2021ZD0110404)
- National Natural Science Foundation of China (62171347)
- Ministry of Education
This publication has 54 references indexed in Scilit:
- U-Net: Convolutional Networks for Biomedical Image Segmentation2015
- EmoNets: Multimodal deep learning approaches for emotion recognition in videoJournal on Multimodal User Interfaces, 2015
- ModDrop: Adaptive Multi-Modal Gesture RecognitionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2015
- Cross-modal Retrieval with Correspondence Autoencoder2014
- Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation2014
- Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation2014
- On the Role of Correlation and Abstraction in Cross-Modal Multimedia RetrievalIEEE Transactions on Pattern Analysis and Machine Intelligence, 2013
- Remote Sensing Image Retrieval With Global Morphological Texture DescriptorsIEEE Transactions on Geoscience and Remote Sensing, 2013
- Indexing of Satellite Images With Different Resolutions by Wavelet FeaturesIEEE Transactions on Image Processing, 2008
- Emotion Recognition through Multiple Modalities: Face, Body Gesture, Speech2008