CLIP-DINOiser: Teaching CLIP a few DINO tricks
Preprint
Abstract
The popular CLIP model displays impressive zero-shot capabilities thanks to its seamless interaction with arbitrary text prompts. However, its lack of spatial awareness makes it unsuitable for dense computer vision tasks, e.g., semantic segmentation, without an additional fine-tuning step that often uses annotations and can potentially suppress its original open-vocabulary properties. Meanwhile, self-supervised representation methods have demonstrated good localization properties without human-made annotations nor explicit supervision. In this work, we take the best of both worlds and propose a zero-shot open-vocabulary semantic segmentation method, which does not require any annotations. We propose to locally improve dense MaskCLIP features, computed with a simple modification of CLIP's last pooling layer, by integrating localization priors extracted from self-supervised features. By doing so, we greatly improve the performance of MaskCLIP and produce smooth outputs. Moreover, we show that the used self-supervised feature properties can directly be learnt from CLIP features therefore allowing us to obtain the best results with a single pass through CLIP model. Our method CLIP-DINOiser needs only a single forward pass of CLIP and two light convolutional layers at inference, no extra supervision nor extra memory and reaches state-of-the-art results on challenging and fine-grained benchmarks such as COCO, Pascal Context, Cityscapes and ADE20k. The code to reproduce our results is available at https://github.com/wysoczanska/clip_dinoiser.Keywords
All Related Versions
This publication has 36 references indexed in Scilit:
- Diffusion Models for Open-Vocabulary Segmentation2024
- Unsupervised Object Localization in the Era of Self-Supervised ViTs: A SurveyInternational Journal of Computer Vision, 2024
- Scaling Open-Vocabulary Image Segmentation with Image-Level Labels2022
- Open-World Semantic Segmentation via Contrasting and Clustering Vision-Language Embedding2022
- Extract Free Dense Labels from CLIP2022
- Self-Supervised Transformers for Unsupervised Object Discovery using Normalized Cut2022
- Decoupling Zero-Shot Semantic Segmentation2022
- Semantic Understanding of Scenes Through the ADE20K DatasetInternational Journal of Computer Vision, 2018
- COCO-Stuff: Thing and Stuff Classes in Context2018
- The Cityscapes Dataset for Semantic Urban Scene Understanding2016