Progressive Learning-to-Learn Knowledge Distillation for Robust Out-of-Domain Generalization
- 4 June 2026
- journal article
- Published by World Scientific Pub Co Pte Ltd in International Journal of Semantic Computing
- Vol. 20 (02) , 227-248
Abstract
Knowledge distillation (KD) plays a crucial role in reducing computational costs, accelerating inference, and improving model generalization. However, when there is a significant capacity gap between teacher and student networks, KD often struggles to transfer knowledge effectively, and robustness under distribution shifts remains a major challenge. To address these issues, we propose learning-to-learn knowledge distillation (L2L-KD), a dynamic temperature-controlled KD framework that progressively increases learning difficulty, mimicking how human learners advance from basic to complex concepts. To further enhance robustness and generalization, we introduce a counterfactual data augmentation technique that leverages the Metropolis–Hastings algorithm to generate fluent and semantically coherent out-of-domain (OOD) samples. We evaluate L2L-KD across in-domain, OOD, and adversarial scenarios, and the results show that it consistently outperforms existing KD approaches while substantially improving robustness. Moreover, building upon this foundation, we extend the core learning philosophy to a new unsupervised cross-domain framework, demonstrating that the dynamic distillation principles of L2L-KD can naturally generalize to broader domain adaptation tasks.Keywords
This publication has 22 references indexed in Scilit:
- A Progressive Learning-to-Learn Knowledge Distillation Method for Out-of-Domain Generalization2025
- MATE-KD: Masked Adversarial TExt, a Companion to Knowledge Distillation2021
- Learning From Multiple Experts: Self-paced Knowledge Distillation for Long-Tailed Classification2020
- M-ADDA: Unsupervised Domain Adaptation with Deep Metric Learning2020
- SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization2020
- Knowledge Distillation via Route Constrained Optimization2019
- Curriculum Dropout2017
- Deep CORAL: Correlation Alignment for Deep Domain Adaptation2016
- EIEACM SIGARCH Computer Architecture News, 2016
- Curriculum learning2009