DF2RQ: Dynamic Feature Fusion via Region-Wise Queries for Semantic Segmentation of Multimodal Remote Sensing Data
- 6 January 2025
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Geoscience and Remote Sensing
- Vol. 63 (01962892) , 1-15
Abstract
Although remote sensing (RS) data with multiple modalities can be used to significantly improve the accuracy of semantic segmentation in RS data, how to effectively extract multimodal information through multimodal feature fusion remains a challenging task. Specifically, existing methods for multimodal feature fusion still face two major challenges: 1) Due to the diverse imaging mechanisms of multimodal RS data, the boundaries of the same foreground may vary across different modalities, leading to the inclusion of unwanted background semantics in the fused foreground features; 2) RS data from different modalities exhibit varying discriminative abilities for different foregrounds, making it challenging to determine the proportion of semantic information for each modality in the fusion results. To address the above issues, we propose a dynamic feature fusion method based on region-wise queries, namely DF 2 RQ, for SS of multimodal RS data. This method is primarily composed of two components: the spatial reconstruction (SR) module and the dynamic fusion (DF) module. Within the SR module, we propose a spatial reconstruction scheme that samples foreground features from different modalities, achieving independent reconstruction of different unimodal features, thereby alleviating the semantic mixing between foreground and background across modalities. In the DF module, a feature fusion scheme based on unimodal feature reference positions is proposed to obtain fusion weights for each modality, thereby enabling the dynamic fusion of complementary features from multiple modalities. The performance of the proposed method has been extensively evaluated on various multimodal RS datasets for SS, and the experimental results consistently show that the proposed method achieves state-of-the-art accuracy on multiple commonly used metrics. In addition, our code is available at https://github.com/I3ab/DF2RQ.Keywords
Funding Information
- National Natural Science Foundation of China (62371140)
- National Key Research and Development Program of China (2022YFB3903404)
This publication has 58 references indexed in Scilit:
- How far are we to GPT-4V? Closing the gap to commercial multimodal models with open-source suitesScience China Information Sciences, 2024
- BEVFormer: Learning Bird’s-Eye-View Representation from Multi-camera Images via Spatiotemporal Transformers2022
- MultiMAE: Multi-modal Multi-task Masked Autoencoders2022
- CMGFNet: A deep cross-modal gated fusion network for building extraction from very high-resolution remote sensing imagesISPRS Journal of Photogrammetry and Remote Sensing, 2022
- ABCNet: Attentive bilateral contextual network for efficient semantic segmentation of Fine-Resolution remotely sensed imageryISPRS Journal of Photogrammetry and Remote Sensing, 2021
- Deep multimodal fusion for semantic image segmentation: A surveyImage and Vision Computing, 2021
- Unified Perceptual Parsing for Scene Understanding2018
- Beyond RGB: Very high resolution urban remote sensing with multimodal deep networksISPRS Journal of Photogrammetry and Remote Sensing, 2018
- FuseNet: Incorporating Depth into Semantic Segmentation via Fusion-Based CNN Architecture2017
- U-Net: Convolutional Networks for Biomedical Image Segmentation2015