Domain Adaptation with a Single Vision-Language Embedding
Preprint
Abstract
Domain adaptation has been extensively investigated in computer vision but still requires access to target data at the training time, which might be difficult to obtain in some uncommon conditions. In this paper, we present a new framework for domain adaptation relying on a single Vision-Language (VL) latent embedding instead of full target data. First, leveraging a contrastive language-image pre-training model (CLIP), we propose prompt/photo-driven instance normalization (PIN). PIN is a feature augmentation method that mines multiple visual styles using a single target VL latent embedding, by optimizing affine transformations of low-level source features. The VL embedding can come from a language prompt describing the target domain, a partially optimized language prompt, or a single unlabeled target image. Second, we show that these mined styles (i.e., augmentations) can be used for zero-shot (i.e., target-free) and one-shot unsupervised domain adaptation. Experiments on semantic segmentation demonstrate the effectiveness of the proposed method, which outperforms relevant baselines in the zero-shot and one-shot settings.Keywords
All Related Versions
This publication has 47 references indexed in Scilit:
- ManiFest: Manifold Deformation for Few-Shot Image Translation2022
- Learning to Prompt for Vision-Language ModelsInternational Journal of Computer Vision, 2022
- Scale-Aware Domain Adaptive Faster R-CNNInternational Journal of Computer Vision, 2021
- Content-Consistent Matching for Domain Adaptive Semantic Segmentation2020
- ICNet for Real-Time Semantic Segmentation on High-Resolution Images2018
- Unsupervised Domain Adaptation for Semantic Segmentation via Class-Balanced Self-training2018
- Semantic Foggy Scene Understanding with Synthetic DataInternational Journal of Computer Vision, 2018
- Playing for Data: Ground Truth from Computer Games2016
- U-Net: Convolutional Networks for Biomedical Image Segmentation2015
- A theory of learning from different domainsMachine Learning, 2009