CoPT: Unsupervised Domain Adaptive Segmentation using Domain-Agnostic Text Embeddings
Preprint
Abstract
Unsupervised domain adaptation (UDA) involves learning class semantics from labeled data within a source domain that generalize to an unseen target domain. UDA methods are particularly impactful for semantic segmentation, where annotations are more difficult to collect than in image classification. Despite recent advances in large-scale vision-language representation learning, UDA methods for segmentation have not taken advantage of the domain-agnostic properties of text. To address this, we present a novel Covariance-based Pixel-Text loss, CoPT, that uses domain-agnostic text embeddings to learn domain-invariant features in an image segmentation encoder. The text embeddings are generated through our LLM Domain Template process, where an LLM is used to generate source and target domain descriptions that are fed to a frozen CLIP model and combined. In experiments on four benchmarks we show that a model trained using CoPT achieves the new state of the art performance on UDA for segmentation. The code can be found at https://github.com/cfmata/CoPT.Keywords
All Related Versions
This publication has 32 references indexed in Scilit:
- HRDA: Context-Aware High-Resolution Domain-Adaptive Semantic Segmentation2022
- Grounding Visual Representations with Texts for Domain Generalization2022
- Prototypical Contrast Adaptation for Domain Adaptive Semantic Segmentation2022
- DAFormer: Improving Network Architectures and Training Strategies for Domain-Adaptive Semantic Segmentation2022
- Prototypical Pseudo Label Denoising and Target Structure Learning for Domain Adaptive Semantic Segmentation2021
- Content-Consistent Matching for Domain Adaptive Semantic Segmentation2020
- Learning to Adapt Structured Output Space for Semantic Segmentation2018
- Playing for Data: Ground Truth from Computer Games2016
- The SYNTHIA Dataset: A Large Collection of Synthetic Images for Semantic Segmentation of Urban Scenes2016
- The Cityscapes Dataset for Semantic Urban Scene Understanding2016