Cognition Transferring and Decoupling for Text-Supervised Egocentric Semantic Segmentation
- 24 December 2024
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 35 (5) , 4144-4157
Abstract
In this paper, we explore a novel Text-supervised Egocentic Semantic Segmentation (TESS) task that aims to assign pixel-level categories to egocentric images weakly supervised by texts from image-level labels. In this task with prospective potential, the egocentric scenes contain dense wearer-object relations and inter-object interference. However, most recent third-view methods leverage the frozen Contrastive Language-Image Pre-training (CLIP) model, which is pre-trained on the semantic-oriented third-view data and lapses in the egocentric view due to the “relation insensitive" problem. Hence, we propose a Cognition Transferring and Decoupling Network (CTDN) that first learns the egocentric wearer-object relations via correlating the image and text. Besides, a Cognition Transferring Module (CTM) is developed to distill the cognitive knowledge from the large-scale pre-trained model to our model for recognizing egocentric objects with various semantics. Based on the transferred cognition, the Foreground-background Decoupling Module (FDM) disentangles the visual representations to explicitly discriminate the foreground and background regions to mitigate false activation areas caused by foreground-background interferential objects during egocentric relation learning. Extensive experiments on four TESS benchmarks demonstrate the effectiveness of our approach, which outperforms many recent related methods by a large margin. Code will be available at https://github.com/ZhaofengSHI/CTDN.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (U23A20286)
- National Natural Science Foundation of China (62301121)
- National Natural Science Foundation of China (62271119)
- China Postdoctoral Science Foundation (2023M740529)
- Postdoctoral Fellowship Program (Grade B) of China Postdoctoral Science Foundation (GZB20240120)
- Independent Research Project of Civil Aviation Flight Technology and Flight Safety Key Laboratory (FZ2022ZZ06)
This publication has 90 references indexed in Scilit:
- Generative Adversarial Network for Future Hand Segmentation from Egocentric Video2022
- Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applications2022
- Content-Consistent Matching for Domain Adaptive Semantic Segmentation2020
- Prototype Mixture Models for Few-Shot Semantic Segmentation2020
- Part-Aware Prototype Network for Few-Shot Semantic Segmentation2020
- Mining Cross-Image Semantics for Weakly Supervised Semantic Segmentation2020
- Weakly-Supervised Semantic Segmentation by Iterative Affinity LearningInternational Journal of Computer Vision, 2020
- Seed, Expand and Constrain: Three Principles for Weakly-Supervised Image Segmentation2016
- Microsoft COCO: Common Objects in Context2014
- The Pascal Visual Object Classes (VOC) ChallengeInternational Journal of Computer Vision, 2009