ReCLIP++: Learn to Rectify the Bias of CLIP for Unsupervised Semantic Segmentation
Preprint
Abstract
Recent works utilize CLIP to perform the challenging unsupervised semantic segmentation task where only images without annotations are available. However, we observe that when adopting CLIP to such a pixel-level understanding task, unexpected bias (including class-preference bias and space-preference bias) occurs. Previous works don't explicitly model the bias, which largely constrains the segmentation performance. In this paper, we propose to explicitly model and rectify the bias existing in CLIP to facilitate the unsupervised semantic segmentation task. Specifically, we design a learnable ''Reference'' prompt to encode class-preference bias and a projection of the positional embedding in vision transformer to encode space-preference bias respectively. To avoid interference, two kinds of biases are firstly independently encoded into the Reference feature and the positional feature. Via a matrix multiplication between two features, a bias logit map is generated to explicitly represent two kinds of biases. Then we rectify the logits of CLIP via a simple element-wise subtraction. To make the rectified results smoother and more contextual, we design a mask decoder which takes the feature of CLIP and rectified logits as input and outputs a rectified segmentation mask with the help of Gumbel-Softmax operation. To make the bias modeling and rectification process meaningful and effective, a contrastive loss based on masked visual features and the text features of different classes is imposed. To further improve the segmentation, we distill the knowledge from the rectified CLIP to the advanced segmentation architecture via minimizing our designed mask-guided, feature-guided and text-guided loss terms. Extensive experiments on various benchmarks demonstrate that ReCLIP++ performs favorably against previous SOTAs. The implementation is available at: https://github.com/dogehhh/ReCLIP.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- VISA: Reasoning Video Object Segmentation via Large Language Models2024
- Learning Neural Eigenfunctions for Unsupervised Semantic Segmentation2023
- Detecting Everything in the Open World: Towards Universal Object Detection2023
- Learning to Prompt for Vision-Language ModelsInternational Journal of Computer Vision, 2022
- Contrastive Adaptation Network for Single- and Multi-Source Domain AdaptationIEEE Transactions on Pattern Analysis and Machine Intelligence, 2020
- DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2017
- SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image SegmentationIEEE Transactions on Pattern Analysis and Machine Intelligence, 2017
- Fully Convolutional Networks for Semantic SegmentationIEEE Transactions on Pattern Analysis and Machine Intelligence, 2016
- The Pascal Visual Object Classes (VOC) ChallengeInternational Journal of Computer Vision, 2009
- The Hungarian method for the assignment problemNaval Research Logistics Quarterly, 1955