Progressive Source-Aware Transformer for Generalized Source-Free Domain Adaptation
- 2 October 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 26 (15209210) , 4138-4152
Abstract
Source-free domain adaptation (SFDA) tends to forget the source domain, suffering from limitations in realworld scenarios. Recently, generalized source-free domain adaptation (GSFDA) problem naturally emerges, aiming for good performance on both target and source domains. The existing methods attempt to retain model parameters associated with the source domain to prevent such forgetting. However, this strategy is not conducive to improving cross-domain performance on the target domain, prioritizing mitigating forgetting on the source domain. This paper introduces a Progressive Source-Aware Transformer approach for GSFDA, dubbed PSAT-GDA. Our core idea is to enforce the domain adaptation process to remember the source domain by imposing source guidance, offering a target domain-centric anti-forgetting mechanism. Specifically, for each epoch, a Transformer-based deep network is adapted to do domain alignment like the traditional SFDA method, because the transformer working on the image patch sequence helps to reduce image noise caused by domain shift. Meanwhile, another Transformer is designed to generate source guidance supervising domain alignment. By augmenting target sample and mining the source information from the historical models before current epoch, source injected feature group is constructed. Based on the Transformer mechanism, the attention block can select useful source information for each target sample. From it, we devise neighbour-based and augmentation-based regularizations to shape the source guidance. Experiments on three challenging datasets show that our method can achieve evident cross-domain improvement on the target domains. Also, it can mitigate forgetting on all domains after adapting to single or multiple target domains. Our resources are available at https://github.com/tntek/PSAT-GDA .Keywords
Funding Information
- Deutsche Forschungsgemeinschaft
- National Natural Science Foundation of China
- Hamburg Landesforschungsförderungsprojekt Cross
- National Natural Science Foundation of China (61773083)
- Horizon2020 RISE (691154)
- National Key R&D Program of China (2020YFB1313600)
- National Natural Science Foundation of China (62276048, 62206168)
This publication has 35 references indexed in Scilit:
- Casting a BAIT for offline and online source-free domain adaptationComputer Vision and Image Understanding, 2023
- Exploring Domain-Invariant Parameters for Source Free Domain Adaptation2022
- Towards Discriminability and Diversity: Batch Nuclear-Norm Maximization Under Label Insufficient Situations2020
- DLOW: Domain Flow for Adaptation and Generalization2019
- AdaGraph: Unifying Predictive and Continuous Domain Adaptation Through Graphs2019
- Task-Free Continual Learning2019
- Deep Clustering for Unsupervised Learning of Visual Features2018
- Adaptive pedestrian detection by predicting classifierNeural Computing and Applications, 2017
- Deep Hashing Network for Unsupervised Domain Adaptation2017
- Deep Residual Learning for Image Recognition2016