Dynamic Distillation Network for Cross-Domain Few-Shot Recognition with Unlabeled Data
Preprint
Abstract
Most existing works in few-shot learning rely on meta-learning the network on a large base dataset which is typically from the same domain as the target dataset. We tackle the problem of cross-domain few-shot learning where there is a large shift between the base and target domain. The problem of cross-domain few-shot recognition with unlabeled target data is largely unaddressed in the literature. STARTUP was the first method that tackles this problem using self-training. However, it uses a fixed teacher pretrained on a labeled base dataset to create soft labels for the unlabeled target samples. As the base dataset and unlabeled dataset are from different domains, projecting the target images in the class-domain of the base dataset with a fixed pretrained model might be sub-optimal. We propose a simple dynamic distillation-based approach to facilitate unlabeled images from the novel/base dataset. We impose consistency regularization by calculating predictions from the weakly-augmented versions of the unlabeled images from a teacher network and matching it with the strongly augmented versions of the same images from a student network. The parameters of the teacher network are updated as exponential moving average of the parameters of the student network. We show that the proposed network learns representation that can be easily adapted to the target domain even though it has not been trained with target-specific classes during the pretraining phase. Our model outperforms the current state-of-the art method by 4.4% for 1-shot and 3.6% for 5-shot classification in the BSCD-FSL benchmark, and also shows competitive performance on traditional in-domain few-shot learning task.Keywords
All Related Versions
This publication has 12 references indexed in Scilit:
- Interpolation consistency training for semi-supervised learningNeural Networks, 2022
- A Broad Study on the Transferability of Visual Representations with Contrastive Learning2021
- Emerging Properties in Self-Supervised Vision Transformers2021
- A Broader Study of Cross-Domain Few-Shot Learning2020
- Rethinking Few-Shot Image Classification: A Good Embedding is All You Need?2020
- EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover ClassificationIEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019
- DeepWeeds: A Multiclass Weed Species Image Dataset for Deep LearningScientific Reports, 2019
- Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised LearningIEEE Transactions on Pattern Analysis and Machine Intelligence, 2018
- Remote Sensing Image Scene Classification: Benchmark and State of the ArtProceedings of the IEEE, 2017
- Human-level concept learning through probabilistic program inductionScience, 2015