Hypergraph-Guided Multimodal Prototype for Remote Sensing Scene Understanding
- 27 January 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Geoscience and Remote Sensing
- Vol. 63 (01962892) , 1-19
Abstract
Noticeable achievements have been made in entity-level perception tasks (e.g., object detection) in remote sensing image interpretation. But for remote sensing images carrying rich content, individual perception cannot well obtain the interaction patterns between entities. The recognition of relationships between entities is the key to deeply understanding remote sensing scenes. In this paper, we propose a hypergraph-guided multimodal prototype network (HMPNet), which performs relation recognition by matching relation representations with multimodal predicate prototypes. To overcome the imbalance of modal information in the matching process, a multimodal calibration strategy is devised, taking into account the image sub-prototype and text sub-prototype, which makes prediction results more reliable. Meanwhile, to align image and text sub-prototypes and explore relevant semantic patterns, the multimodal hypergraph is constructed to efficiently capture the associations between heterogeneous prototypes. Experimental results show that the performance of our model can reach the state-of-the-art (SOTA) level on the remote sensing scene graph generation task.Keywords
Funding Information
- National Key Research and Development Program of China (2021YFB3900504)
- National Natural Science Foundation of China (42301437)
This publication has 78 references indexed in Scilit:
- Knowledge-guided land pattern depiction for urban land use mapping: A case study of Chinese citiesRemote Sensing of Environment, 2022
- FAIR1M: A benchmark dataset for fine-grained object recognition in high-resolution remote sensing imageryISPRS Journal of Photogrammetry and Remote Sensing, 2022
- Hypergraph convolution and hypergraph attentionPattern Recognition, 2021
- Object detection in optical remote sensing images: A survey and a new benchmarkISPRS Journal of Photogrammetry and Remote Sensing, 2020
- Graph R-CNN for Scene Graph Generation2018
- Scene Graph Generation by Iterative Message Passing2017
- Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image AnnotationsInternational Journal of Computer Vision, 2017
- Visual Relationship Detection with Language Priors2016
- Deep Residual Learning for Image Recognition2016
- Image retrieval using scene graphs2015