Injecting Linguistic Into Visual Backbone: Query-Aware Multimodal Fusion Network for Remote Sensing Visual Grounding
- 26 August 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Geoscience and Remote Sensing
- Vol. 62 (01962892) , 1-14
Abstract
The Remote Sensing Visual Grounding (RSVG) task focuses on accurately identifying and localizing specific targets in remote sensing (RS) images using descriptive query expressions. Existing methods independently extract visual and textual features, ignoring early complementary information between image and text. This leads to information loss and misalignment, limiting the model’s ability to distinguish similar targets. To address this challenge, we propose the Query-Aware Multimodal Fusion Network (QAMFN), which introduces an innovative Query-Guided Visual Attention (QGVA) mechanism in the early stages of the visual encoder. This mechanism integrates textual information during the early visual feature extraction process, thereby resolving the issue of missing image-text complementary information. QGVA ensures that the visual backbone accurately focuses on local features highly relevant to the query by injecting textual information into the visual encoding process. Additionally, to enhance the model’s ability to integrate multimodal information and adapt to more complex RS images, we introduce the Text-Semantic Attention-Guided Masking (TAM) module. TAM aggregates multimodal features processed by the backbones and filters out redundant information, producing high-quality fused features. Experiments demonstrate that our approach sets a new record on the DIOR-RSVG dataset, improving accuracy to 81.67% (an absolute increase of 4.98%).Keywords
Funding Information
- National Key Research and Development Program of China (2022ZD0118402)
- National Natural Science Foundation of China (62331027)
This publication has 45 references indexed in Scilit:
- From single- to multi-modal remote sensing imagery interpretation: a survey and taxonomyScience China Information Sciences, 2023
- SeqTR: A Simple Yet Universal Network for Visual Grounding2022
- Improving One-Stage Visual Grounding by Recursive Sub-query Construction2020
- End-to-End Object Detection with Transformers2020
- Object detection in optical remote sensing images: A survey and a new benchmarkISPRS Journal of Photogrammetry and Remote Sensing, 2020
- MAttNet: Modular Attention Network for Referring Expression Comprehension2018
- Generalized Semantic Preserving Hashing for N-Label Cross-Modal Retrieval2017
- Deep Residual Learning for Image Recognition2016
- Remote sensing big data computing: Challenges and opportunitiesFuture Generation Computer Systems, 2015
- Visualizing and Understanding Convolutional Networks2014