Transferability-Guided Cross-Domain Cross-Task Transfer Learning
- 5 February 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks and Learning Systems
- Vol. 36 (2) , 2423-2436
Abstract
We propose two novel transferability metrics fast optimal transport-based conditional entropy (F-OTCE) and joint correspondence OTCE (JC-OTCE) to evaluate how much the source model (task) can benefit the learning of the target task and to learn more generalizable representations for cross-domain cross-task transfer learning. Unlike the original OTCE metric that requires evaluating the empirical transferability on auxiliary tasks, our metrics are auxiliary-free such that they can be computed much more efficiently. Specifically, F-OTCE estimates transferability by first solving an optimal transport (OT) problem between source and target distributions and then uses the optimal coupling to compute the negative conditional entropy (NCE) between the source and target labels. It can also serve as an objective function to enhance downstream transfer learning tasks including model finetuning and domain generalization (DG). Meanwhile, JC-OTCE improves the transferability accuracy of F-OTCE by including label distances in the OT problem, though it incurs additional computation costs. Extensive experiments demonstrate that F-OTCE and JC-OTCE outperform state-of-the-art auxiliary-free metrics by $21.1\%$ and $25.8\%$ , respectively, in correlation coefficient with the ground-truth transfer accuracy. By eliminating the training cost of auxiliary tasks, the two metrics reduce the total computation time of the previous method from 43 min to 9.32 and 10.78 s, respectively, for a pair of tasks. When applied in the model finetuning and DG tasks, F-OTCE shows significant improvements in the transfer accuracy in few-shot classification experiments, with up to $4.41\%$ and $2.34\%$ accuracy gains, respectively.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (62371270)
- Tsinghua Shenzhen International Graduate School (SIGS) Scientific Research Start-Up Fund (QD2021012C)
- Shenzhen Key Laboratory of Ubiquitous Data Enabling (ZDSYS20220527171406015)
This publication has 38 references indexed in Scilit:
- Meta-Transfer Learning for Few-Shot Learning2019
- Rotation Equivariant CNNs for Digital Pathology2018
- Taskonomy: Disentangling Task Transfer Learning2018
- Learning to Compare: Relation Network for Few-Shot Learning2018
- Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles2016
- Deep Residual Learning for Image Recognition2016
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015
- Describing Textures in the Wild2014
- A theory of learning from different domainsMachine Learning, 2009
- Transfer bounds for linear feature learningMachine Learning, 2009