Abstract
In-context segmentation, also known as one-shot segmentation, aims to segment objects based on a single labeled example. While the Segment Anything Model (SAM) excels in interactive segmentation, it is not inherently designed for in-context tasks. To bridge this gap, we propose a new Dual Consistency SAM (DC-SAM), a prompt-tuning framework that adapts SAM and SAM2 for both image and video in-context segmentation. Instead of relying solely on pre-trained backbones, DC-SAM enhances the prompt encoder by generating high-quality visual prompts through feature fusion. Furthermore, we introduce a novel cycle-consistent cross-attention mechanism to enforce alignment between fused features and visual prompts, complemented by a dual-branch design incorporating discriminative positive and negative prompts. Additionally, we extend DC-SAM to the video domain via a novel mask-tube training strategy. To facilitate research, we curate the first In-Context Video Object Segmentation (IC-VOS) benchmark. Extensive experiments demonstrate that DC-SAM achieves state-of-the-art performance, yielding 55.5 mIoU (+1.4) on COCO-20$^{i}$, 73.0 (+1.1) mIoU on PASCAL-5$^{i}$, and a $\mathcal {J} { \& amp;} \mathcal{F}$ score of 71.52 on IC-VOS.
Funding Information
  • National Natural Science Foundation of China (62572072)
  • National Natural Science Foundation of China (U24B20176)
  • Natural Science Foundation of Beijing Municipality (L243027)