Vision and Language Reference Prompt into SAM for Few-shot Segmentation
Preprint
Abstract
Segment Anything Model (SAM) represents a large-scale segmentation model that enables powerful zero-shot capabilities with flexible prompts. While SAM can segment any object in zero-shot, it requires user-provided prompts for each target image and does not attach any label information to masks. Few-shot segmentation models addressed these issues by inputting annotated reference images as prompts to SAM and can segment specific objects in target images without user-provided prompts. Previous SAM-based few-shot segmentation models only use annotated reference images as prompts, resulting in limited accuracy due to a lack of reference information. In this paper, we propose a novel few-shot segmentation model, Vision and Language reference Prompt into SAM (VLP-SAM), that utilizes the visual information of the reference images and the semantic information of the text labels by inputting not only images but also language as reference information. In particular, VLP-SAM is a simple and scalable structure with minimal learnable parameters, which inputs prompt embeddings with vision-language information into SAM using a multimodal vision-language model. To demonstrate the effectiveness of VLP-SAM, we conducted experiments on the PASCAL-5i and COCO-20i datasets, and achieved high performance in the few-shot segmentation task, outperforming the previous state-of-the-art model by a large margin (6.3% and 9.5% in mIoU, respectively). Furthermore, VLP-SAM demonstrates its generality in unseen objects that are not included in the training data. Our code is available at https://github.com/kosukesakurai1/VLP-SAM.Keywords
All Related Versions
This publication has 14 references indexed in Scilit:
- LARE: Latent augmentation using regional embedding with vision-language modelMachine Learning with Applications, 2025
- Foreground-Covering Prototype Generation and Matching for SAM-Aided Few-Shot SegmentationProceedings of the AAAI Conference on Artificial Intelligence, 2025
- Explore In-Context Segmentation via Latent Diffusion ModelsProceedings of the AAAI Conference on Artificial Intelligence, 2025
- Partial visual-semantic embedding: Fine-grained outfit image representation with massive volumes of tags via angular-based contrastive learningKnowledge-Based Systems, 2023
- SAMRS: Scaling-up Remote Sensing Segmentation Dataset with Segment Anything Model2023
- Fashion intelligence system: An outfit interpretation utilizing images and rich abstract tagsExpert Systems with Applications, 2022
- Learning to Prompt for Vision-Language ModelsInternational Journal of Computer Vision, 2022
- CLIP4Clip: An empirical study of CLIP for end to end video clip retrieval and captioningNeurocomputing, 2022
- Wav2CLIP: Learning Robust Audio Representations from Clip2022
- Backpropagation Applied to Handwritten Zip Code RecognitionNeural Computation, 1989