Decoupling Zero-Shot Semantic Segmentation
Authors:
Top Cited Papers
- 1 June 2022
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 11573-11582
Abstract
Zero-shot semantic segmentation (ZS3) aims to segment the novel categories that have not been seen in the training. Existing works formulate ZS3 as a pixel-level zeroshot classification problem, and transfer semantic knowledge from seen classes to unseen ones with the help of language models pre-trained only with texts. While simple, the pixel-level ZS3 formulation shows the limited capability to integrate vision-language models that are often pre-trained with image-text pairs and currently demonstrate great potential for vision tasks. Inspired by the observation that humans often perform segment-level semantic labeling, we propose to decouple the ZS3 into two sub-tasks: 1) a classagnostic grouping task to group the pixels into segments. 2) a zero-shot classification task on segments. The former task does not involve category information and can be directly transferred to group pixels for unseen classes. The latter task performs at segment-level and provides a natural way to leverage large-scale vision-language models pre-trained with image-text pairs (e.g. CLIP) for ZS3. Based on the decoupling formulation, we propose a simple and effective zero-shot semantic segmentation model, called ZegFormer, which outperforms the previous methods on ZS3 standard benchmarks by large margins, e.g., 22 points on the PAS-CAL VOC and 3 points on the COCO-Stuff in terms of mIoU for unseen classes. Code will be released at https://github.com/dingjiansw101/ZegFormer.Keywords
All Related Versions
This publication has 31 references indexed in Scilit:
- Multiscale Combinatorial Grouping2014
- COSTA: Co-Occurrence Statistics for Zero-Shot Classification2014
- Attribute-Based Classification for Zero-Shot Visual Object CategorizationIEEE Transactions on Pattern Analysis and Machine Intelligence, 2013
- Semantic Label Sharing for Learning with Many Categories2010
- The Pascal Visual Object Classes (VOC) ChallengeInternational Journal of Computer Vision, 2009
- Image segmentation by data-driven markov chain monte carloIEEE Transactions on Pattern Analysis and Machine Intelligence, 2002
- Normalized cuts and image segmentationIEEE Transactions on Pattern Analysis and Machine Intelligence, 2000
- Region competition: unifying snakes, region growing, and Bayes/MDL for multiband image segmentationIEEE Transactions on Pattern Analysis and Machine Intelligence, 1996
- WordNetCommunications of the ACM, 1995
- Recognition-by-components: A theory of human image understanding.Psychological Review, 1987