Multimodal Visual-Language Prompt Network for Remote Sensing Few-Shot Segmentation
- 4 July 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Geoscience and Remote Sensing
- Vol. 63 (01962892) , 1-16
Abstract
Few-shot segmentation (FSS) aims to segment objects of interest in a query image using a limited set of support images. However, most existing FSS methods are designed for natural images. When extended to remote sensing scenes characterized by extreme intra-class variations and complex backgrounds, these methods struggle to provide robust segmentation guidance, leading to severe performance degradation. To address the aforementioned issues, we propose a multimodal visual-language prompt network (MVLPNet), which employs a collaborative optimization strategy for visual-textual features to tackle the remote sensing FSS task. Specifically, MVLPNet consists of a textual-visual consistency enhancement (TVCE) module and a prototype-guided semantic alignment (PGSA) module. To overcome the limited support set for better guiding the query segmentation, we propose a TVCE module that leverages the contrastive language-image pre-training model (CLIP) to capture category-specific text embeddings. An optimal transport (OT) plan is then established to tightly align these text embeddings with the visual features of query image, thereby extracting semantic information from the query image itself to mitigate the extreme intra-class variation in remote sensing images. Furthermore, a PGSA module is proposed to suppress interference caused by complex background regions. By aggregating lost foreground regions, more comprehensive support features are extracted. Then, the query and support features are precisely matched to activate consistent foreground regions, rather than ambiguously matching the query features via a single prototype or multiple prototypes. Extensive experiments on the iSAID-5i and LoveDA-2i datasets have demonstrated that our method achieves the state of the art. The code is available https://github.com/Gritiii/MVLPNet.Keywords
Funding Information
- Beijing Natural Science Foundation (4232022)
- North China University of Technology (NCUT) through the Yuxiu Innovation Project (2024NCUTYXCX119, 2024NCUTYXCX201)
This publication has 67 references indexed in Scilit:
- AgMTR: Agent Mining Transformer for Few-Shot Segmentation in Remote SensingInternational Journal of Computer Vision, 2024
- Self-support Few-Shot Semantic Segmentation2022
- Prototype Mixture Models for Few-Shot Semantic Segmentation2020
- Object detection in optical remote sensing images: A survey and a new benchmarkISPRS Journal of Photogrammetry and Remote Sensing, 2020
- Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based LocalizationInternational Journal of Computer Vision, 2019
- Algorithms for semantic segmentation of multispectral remote sensing imagery using deep learningISPRS Journal of Photogrammetry and Remote Sensing, 2018
- Pyramid Scene Parsing Network2017
- Deep Residual Learning for Image Recognition2016
- U-Net: Convolutional Networks for Biomedical Image Segmentation2015
- Fully convolutional networks for semantic segmentation2015