Enhancing Remote Sensing Visual Question Answering: A Mask-Based Dual-Stream Feature Mutual Attention Network
- 16 April 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Geoscience and Remote Sensing Letters
- Vol. 21 (1545598X) , 1-5
Abstract
The visual question answering (VQA) method applied to remote sensing images (RSIs) can complete the interaction of image information and text information, which avoids professional barriers in different RSIs processing fields. The current methods face challenges in both fully using the global and local information of the image to interact with the question information and addressing the issue of interclass interference. To address these challenges, this letter proposes a remote sensing visual question answering (RSVQA) mask-based dual-stream feature mutual attention network (MADNet). First, the dual-stream feature extraction module of the image is used to obtain image features, and the deep and shallow layer feature encoding module is used to obtain question features. Second, the attention mechanism is introduced and combined with the pointwise multiplication method to use the dual-stream features that were extracted in the earlier step. Finally, an answer relevance modulation module based on a binary mask vector is implemented to filter out irrelevant answers. In the experiments, the performance of the proposed strategy is evaluated using two datasets collected by aerial and Sentinel-2 sensors. In our study, we propose a model that outperforms previous approaches, achieving a 6.89% increase in overall accuracy (OA) over the baseline. This enhancement is notable for its persistence, even when the training data are reduced by half, as evidenced by our experiments on the low-resolution (LR) dataset.Keywords
Funding Information
- Research Project of SongShan Laboratory (YYJC052022004)
- National Natural Science Foundation of China (62101517)
- National Natural Science Foundation of China (62176200)
- Natural Science Basic Research Program of Shaanxi (2022JC-45)
- Fund for Foreign Scholars in University Research and Teaching Programs
This publication has 19 references indexed in Scilit:
- Temporal difference-guided network for hyperspectral image change detectionInternational Journal of Remote Sensing, 2023
- Grid Network: Feature Extraction in Anisotropic Perspective for Hyperspectral Image ClassificationIEEE Geoscience and Remote Sensing Letters, 2023
- RSVQA: Visual Question Answering for Remote Sensing DataIEEE Transactions on Geoscience and Remote Sensing, 2020
- Deep Modular Co-Attention Networks for Visual Question Answering2019
- Learning Spectral-Spatial-Temporal Features via a Recurrent Convolutional Neural Network for Change Detection in Multispectral ImageryIEEE Transactions on Geoscience and Remote Sensing, 2018
- Deep Learning Classification of Land Cover and Crop Types Using Remote Sensing DataIEEE Geoscience and Remote Sensing Letters, 2017
- Automatic Road Detection and Centerline Extraction via Cascaded End-to-End Convolutional Neural NetworkIEEE Transactions on Geoscience and Remote Sensing, 2017
- Deep Residual Learning for Image Recognition2016
- VQA: Visual Question Answering2015
- Long Short-Term Memory2012