Learning to Retain while Acquiring: Combating Distribution-Shift in Adversarial Data-Free Knowledge Distillation
Authors:
- 1 June 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 7786-7794
Abstract
Data-free Knowledge Distillation (DFKD) has gained popularity recently, with the fundamental idea of carrying out knowledge transfer from a Teacher neural network to a Student neural network in the absence of training data. However, in the Adversarial DFKD framework, the student network's accuracy, suffers due to the non-stationary distribution of the pseudo-samples under multiple generator updates. To this end, at every generator update, we aim to maintain the student's performance on previously encountered examples while acquiring knowledge from samples of the current distribution. Thus, we propose a meta-learning inspired framework by treating the task of Knowledge-Acquisition (learning from newly generated samples) and Knowledge-Retention (retaining knowledge on previously met samples) as meta-train and meta-test, respectively. Hence, we dub our method as Learning to Retain while Acquiring. Moreover, we identify an implicit aligning factor between the Knowledge-Retention and Knowledge-Acquisition tasks indicating that the proposed student update strategy enforces a common gradient direction for both tasks, alleviating interference between the two objectives. Finally, we support our hypothesis by exhibiting extensive evaluation and comparison of our method with prior arts on multiple datasets.Keywords
All Related Versions
This publication has 13 references indexed in Scilit:
- Preventing Catastrophic Forgetting and Distribution Mismatch in Knowledge Distillation via Synthetic Data2022
- Dual discriminator adversarial distillation for data-free model compressionInternational Journal of Machine Learning and Cybernetics, 2021
- Zero-shot Adversarial Quantization2021
- Effectiveness of Arbitrary Transfer Sets for Data-free Knowledge Distillation2021
- Catastrophic forgetting and mode collapse in GANs2020
- DeGAN: Data-Enriching GAN for Retrieving Representative Samples from a Trained ClassifierProceedings of the AAAI Conference on Artificial Intelligence, 2020
- An Introduction to Variational AutoencodersFoundations and Trends in Machine Learning, 2019
- Gradient Matching Generative Networks for Zero-Shot Learning2019
- A Review on Bilevel Optimization: From Classical to Evolutionary Approaches and ApplicationsIEEE Transactions on Evolutionary Computation, 2017
- Deep Residual Learning for Image Recognition2016