Vision and Language Reference for a Segment Anything Model for Few-Shot Segmentation
Top Cited Papers
Open Access
- 24 March 2026
- journal article
- research article
- Published by MDPI AG in Journal of Imaging
- Vol. 12 (4) , 143
Abstract
Segment Anything Model (SAM)-based few-shot segmentation models traditionally rely solely on annotated reference images as prompts, which inherently limits their accuracy due to an over-reliance on visual cues and a lack of semantic context. This reliance leads to incorrect segmentation, where visually similar objects from different categories are incorrectly identified as the target object. We propose Vision and Language Reference Prompt into SAM (VLP-SAM), a novel few-shot segmentation model that integrates both visual information of reference images and semantic information of text labels into SAM. VLP-SAM introduces a vision-language model (VLM) with pixel–text matching into the prompt encoder for SAM, effectively leveraging textual semantic consistency while preserving SAM’s extensive segmentation knowledge. By incorporating task-specific structures such as an attention mask, our model achieves superior few-shot segmentation performance with only 1.4 M learnable parameters. Evaluations on PASCAL-5i and COCO-20i datasets demonstrate that VLP-SAM significantly outperforms previous methods by 6.8% and 9.3% in mIoU, respectively. Furthermore, VLP-SAM exhibits strong generalization across unseen objects and cross-domain scenarios, highlighting the robustness provided by textual semantic guidance. This study offers an effective and scalable framework for few-shot segmentation with multimodal prompts.Keywords
All Related Versions
Funding Information
- Japan Society for the Promotion of Science (21H04600)
- Japan Society for the Promotion of Science (24H00370)
This publication has 61 references indexed in Scilit:
- FS-SAM2: Adapting Segment Anything Model 2 for Few-Shot Semantic Segmentation via Low-Rank Adaptation2026
- LARE: Latent augmentation using regional embedding with vision-language modelMachine Learning with Applications, 2025
- Segment and Recognize Anything at Any Granularity2024
- Grounding DINO: Marrying DINO with Grounded Pre-training for Open-Set Object Detection2024
- IRSAM: Advancing Segment Anything Model for Infrared Small Target Detection2024
- SegIC: Unleashing the Emergent Correspondence for In-Context Segmentation2024
- Simple Open-Vocabulary Object Detection2022
- Learning to Prompt for Vision-Language ModelsInternational Journal of Computer Vision, 2022
- CLIP4Clip: An empirical study of CLIP for end to end video clip retrieval and captioningNeurocomputing, 2022
- Microsoft COCO: Common Objects in Context2014