MENSA: Multi-Dataset Harmonized Pretraining for Semantic Segmentation
- 25 December 2024
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 27 (15209210) , 2127-2140
Abstract
Existing pretraining methods for semantic segmentation are hampered by the task gap between global image-level pretraining and local pixel-level finetuning. Joint dense-level pretraining is a promising alternative to exploit off-the-shelf annotations from diverse segmentation datasets but suffers from low-quality class embeddings and inconsistent data and supervision signals across multiple datasets by directly employing CLIP. To overcome these challenges, we propose a novel M ulti-datas E t harmo N ized pretraining framework for S emantic s E gmentation (MENSA). MENSA incorporates high-quality language embeddings and momentum-updated visual embeddings to effectively model the class relationships in the embedding space and thereby provide reliable supervision information for each category. To further adapt to multiple datasets, we achieve one-to-many pixel-embedding pairing with cross-dataset multi-label mapping through cross-modal information exchange to mitigate inconsistent supervision signals and introduce region-level and pixel-level cross-dataset mixing for varying data distribution. Experimental results demonstrate that MENSA is a powerful foundation segmentation model that consistently outperforms popular supervised or unsupervised ImageNet pretrained models for various benchmarks under standard fine-tuning. Furthermore, MENSA is shown to significantly benefit frozen-backbone fine-tuning and zero-shot learning by endowing pixel-level distinctiveness to learned representations.Keywords
Funding Information
- National Natural Science Foundation of China (62125109, 61931023, 61932022, 62371288, 62320106003, 62120106007)
This publication has 40 references indexed in Scilit:
- Open Vocabulary Object Detection with Pseudo Bounding-Box Labels2022
- Scaling Open-Vocabulary Image Segmentation with Image-Level Labels2022
- A Transformer-Based Decoder for Semantic Segmentation with Multi-level Context Mining2022
- Learning Semantic Segmentation from Multiple Datasets with Label Shifts2022
- Learning to Prompt for Vision-Language ModelsInternational Journal of Computer Vision, 2022
- Semantic Understanding of Scenes Through the ADE20K DatasetInternational Journal of Computer Vision, 2018
- Unsupervised Domain Adaptation for Semantic Segmentation via Class-Balanced Self-training2018
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015
- The Pascal Visual Object Classes Challenge: A RetrospectiveInternational Journal of Computer Vision, 2014
- Microsoft COCO: Common Objects in Context2014