SemiCD-VL: Visual-Language Model Guidance Makes Better Semi-Supervised Change Detector
- 9 December 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Geoscience and Remote Sensing
- Vol. 63 (01962892) , 1-13
Abstract
Change detection (CD) aims to identify pixels with semantic changes between images. However, annotating massive numbers of pixel-level images is labor-intensive and costly, especially for multitemporal images, which require pixel-wise comparisons by human experts. Considering the excellent performance of visual-language models (VLMs) for zero-shot, OV, etc., with prompt-based reasoning, it is promising to utilize VLMs to make better CD under limited labeled data. In this article, we propose a VLM guidance-based semi-supervised CD method, namely SemiCD-VL. The insight of SemiCD-VL is to synthesize free change labels using VLMs to provide additional supervision signals for unlabeled data. However, almost all current VLMs are designed for single-temporal images and cannot be directly applied to bi- or multitemporal images. Motivated by this, we first propose a VLM-based mixed change event generation (CEG) strategy to yield pseudo-labels for unlabeled CD data. Since the additional supervised signals provided by these VLM-driven pseudo-labels may conflict with the original pseudo-labels from the consistency regularization paradigm (e.g., FixMatch), we propose the dual projection head for de-entangling different signal sources. Further, we explicitly decouple the bitemporal images semantic representation through two auxiliary segmentation decoders, which are also guided by VLM. Finally, to make the model more adequately capture change representations, we introduce contrastive consistency regularization (CCR) by constructing feature-level contrastive loss in auxiliary branches. Extensive experiments show the advantage of SemiCD-VL. For instance, SemiCD-VL improves the FixMatch baseline by $+ 5.3~\text {IoU}^{c}$ on WHU-CD and by $+ 2.4~\text {IoU}^{c}$ on LEVIR-CD with 5% labels, and SemiCD-VL requires only 5%–10% of the labels to achieve performance similar to the supervised methods. In addition, our CEG strategy, in an unsupervised manner, can achieve performance far superior to state-of-the-art (SOTA) unsupervised CD methods (e.g., IoU improved from 18.8% to 46.3% on LEVIR-CD dataset). The code is available at https://github.com/likyoo/SemiCD-VL .Keywords
All Related Versions
Funding Information
- National Key Research and Development Program of China (2021ZD0112902)
- China NSFC projects (62272375, 12226004)
This publication has 48 references indexed in Scilit:
- Temporal-agnostic change region proposal for semantic change detectionISPRS Journal of Photogrammetry and Remote Sensing, 2023
- Scaling Open-Vocabulary Image Segmentation with Image-Level Labels2022
- A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-Language Model2022
- Extract Free Dense Labels from CLIP2022
- ChangeMask: Deep multi-task encoder-transformer-decoder architecture for semantic change detectionISPRS Journal of Photogrammetry and Remote Sensing, 2022
- Guided Collaborative Training for Pixel-Wise Semi-Supervised Learning2020
- End-to-End Object Detection with Transformers2020
- ChangeNet: A Deep Learning Architecture for Visual Change Detection2019
- Street-view change detection with deconvolutional networksAutonomous Robots, 2018
- Topological structural analysis of digitized binary images by border followingComputer Vision, Graphics, and Image Processing, 1985