MCTformer+: Multi-Class Token Transformer for Weakly Supervised Semantic Segmentation
Top Cited Papers
- 23 May 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 46 (12) , 8380-8395
Abstract
This paper proposes a novel transformer-based framework to generate accurate class-specific object localization maps for weakly supervised semantic segmentation (WSSS). Leveraging the insight that the attended regions of the one-class token in the standard vision transformer can generate class-agnostic localization maps, we investigate the transformer's capacity to capture class-specific attention for class-discriminative object localization by learning multiple class tokens. We present the Multi-Class Token transformer, which incorporates multiple class tokens to enable class-aware interactions with patch tokens. This is facilitated by a class-aware training strategy that establishes a one-to-one correspondence between output class tokens and ground-truth class labels. We also introduce a Contrastive-Class-Token (CCT) module to enhance the learning of discriminative class tokens, enabling the model to better capture the unique characteristics of each class. Consequently, the proposed framework effectively generates class-discriminative object localization maps from the class-to-patch attentions associated with different class tokens. To refine these localization maps, we propose the utilization of patch-level pairwise affinity derived from the patch-to-patch transformer attention. Furthermore, the proposed framework seamlessly complements the Class Activation Mapping (CAM) method, yielding significant improvements in WSSS performance on PASCAL VOC 2012 and MS COCO 2014. These results underline the importance of the class token for WSSS.Keywords
Funding Information
- ARC (DP210101682, DP210102674, LP220100390, IH180100002)
- UWA (RCA 2024)
- Early Career Scheme of the Research Grants Council (26202321)
- ARC (DP22010219)
This publication has 77 references indexed in Scilit:
- Bagging Regional Classification Activation Maps for Weakly Supervised Object Localization2022
- Adversarial Erasing Framework via Triplet with Gated Pyramid Pooling Layer for Weakly Supervised Semantic Segmentation2022
- Splitting Vs. Merging: Mining Object Regions with Discrepancy and Intersection Loss for Weakly Supervised Semantic Segmentation2020
- Rethinking Class Activation Mapping for Weakly Supervised Object Localization2020
- Weakly Supervised Semantic Segmentation with Boundary Exploration2020
- Mining Cross-Image Semantics for Weakly Supervised Semantic Segmentation2020
- On Regularized Losses for Weakly-supervised CNN Segmentation2018
- Seed, Expand and Constrain: Three Principles for Weakly-Supervised Image Segmentation2016
- Microsoft COCO: Common Objects in Context2014
- The Pascal Visual Object Classes (VOC) ChallengeInternational Journal of Computer Vision, 2009