PointSAM: Pointly-Supervised Segment Anything Model for Remote Sensing Images

Authors:
Top Cited Papers
Publisher website
Google Scholar
Abstract
Segment Anything Model (SAM) is an advanced foundational model for image segmentation, which is gradually being applied to remote sensing images (RSIs). Due to the domain gap between RSIs and natural images, traditional methods typically use SAM as a source pre-trained model and fine-tune it with fully supervised masks. Unlike these methods, our work focuses on fine-tuning SAM using more convenient and challenging point annotations. Leveraging SAM’s zero-shot capability, we adopt a self-training framework that iteratively generates pseudo-labels. However, noisy labels in pseudo-labels can cause error accumulation. To address this, we introduce Prototype-based Regularization, where target prototypes are extracted from the dataset and matched to predicted prototypes using the Hungarian algorithm to guide learning in the correct direction. Additionally, RSIs have complex backgrounds and densely packed objects, making it possible for point prompts to mistakenly group multiple objects as one. To resolve this, we propose a Negative Prompt Calibration method, based on the non-overlapping nature of instance masks, where overlapping masks are used as negative signals to refine segmentation. Combining these techniques, we present a novel pointly-supervised segment anything model, PointSAM. We conduct experiments on three RSI datasets, including WHU, HRSID, and NWPU VHR-10, showing that our method significantly outperforms direct testing with SAM, SAM2, and other comparison methods. Additionally, PointSAM can act as a point-to-box converter for oriented object detection, achieving promising results and indicating its potential for other point-supervised tasks. The code is available at https://github.com/Lans1ng/PointSAM.
Funding Information
  • National Natural Science Foundation of China (62271418, 62106078)
  • Natural Science Foundation of Sichuan Province (23NSFSC0030)
  • China Scholarship Council (CSC202207000056)

This publication has 55 references indexed in Scilit: