Cross-Task Transfer for Multimodal Aerial Scene Recognition
Preprint
Abstract
Aerial scene recognition is a fundamental task in remote sensing and has recently received increased interest. While the visual information from overhead images with powerful models and efficient algorithms yields good performance on scene recognition, additional information is always a bonus, for example, the corresponding audio information. In this paper, for improving the performance on the aerial scene recognition, we explore a novel audiovisual aerial scene recognition task using both images and sounds as input. Based on an observation that some specific sound events are more likely to be heard at a given geographic location, we propose to exploit the knowledge from the sound events to improve the performance on the aerial scene recognition. For this purpose, we have constructed a new dataset named AuDio Visual Aerial sceNe reCognition datasEt (ADVANCE). With the help of this dataset, we evaluate three proposed approaches for transferring the sound event knowledge to the aerial scene recognition task in a multimodal learning framework, and show the benefit of exploiting the audio information for the aerial scene recognition. The source code is publicly available for reproducibility purposes.https://github.com/DTaoo/Multimodal-Aerial-Scene-Recognition.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- Sound Event Detection by Multitask Learning of Sound Events and Scenes with Soft Scene Labels2020
- Learning Sight from Sound: Ambient Sound Provides Supervision for Visual LearningInternational Journal of Computer Vision, 2018
- A Multimodal Approach to Mapping Soundscapes2018
- EmotionMeter: A Multimodal Framework for Recognizing Human EmotionsIEEE Transactions on Cybernetics, 2018
- When Deep Learning Meets Metric Learning: Remote Sensing Image Scene Classification via Learning Discriminative CNNsIEEE Transactions on Geoscience and Remote Sensing, 2018
- AID: A Benchmark Data Set for Performance Evaluation of Aerial Scene ClassificationIEEE Transactions on Geoscience and Remote Sensing, 2017
- Towards better exploiting convolutional neural networks for remote sensing scene classificationPattern Recognition, 2017
- Scene Classification via a Gradient Boosting Random Convolutional Network FrameworkIEEE Transactions on Geoscience and Remote Sensing, 2015
- Deep Learning Based Feature Selection for Remote Sensing Scene ClassificationIEEE Geoscience and Remote Sensing Letters, 2015
- Comparing SIFT descriptors and gabor texture features for classification of remote sensed imagery2008