Dynamic Data-Free Knowledge Distillation by Easy-to-Hard Learning Strategy
Preprint
Abstract
Data-free knowledge distillation (DFKD) is a widely-used strategy for Knowledge Distillation (KD) whose training data is not available. It trains a lightweight student model with the aid of a large pretrained teacher model without any access to training data. However, existing DFKD methods suffer from inadequate and unstable training process, as they do not adjust the generation target dynamically based on the status of the student model during learning. To address this limitation, we propose a novel DFKD method called CuDFKD. It teaches students by a dynamic strategy that gradually generates easy-to-hard pseudo samples, mirroring how humans learn. Besides, CuDFKD adapts the generation target dynamically according to the status of student model. Moreover, We provide a theoretical analysis of the majorization minimization (MM) algorithm and explain the convergence of CuDFKD. To measure the robustness and fidelity of DFKD methods, we propose two more metrics, and experiments shows CuDFKD has comparable performance to state-of-the-art (SOTA) DFKD methods on all datasets. Experiments also present that our CuDFKD has the fastest convergence and best robustness over other SOTA DFKD methods.Keywords
All Related Versions
This publication has 16 references indexed in Scilit:
- Online adversarial knowledge distillation for graph neural networksExpert Systems with Applications, 2023
- Target layer regularization for continual learning using Cramer-Wold distanceInformation Sciences, 2022
- Spaks: Self-paced multiple kernel subspace clustering with feature smoothing regularizationKnowledge-Based Systems, 2022
- Curriculum Learning: A SurveyInternational Journal of Computer Vision, 2022
- Self-Paced Co-Training of Graph Neural Networks for Semi-Supervised Node ClassificationIEEE Transactions on Neural Networks and Learning Systems, 2022
- Unsupervised feature selection via self-paced learning and low-redundant regularizationKnowledge-Based Systems, 2022
- Distilling from professors: Enhancing the knowledge distillation of teachersInformation Sciences, 2021
- Data-Free Model Extraction2021
- On Convergence Properties of Implicit Self-paced ObjectiveInformation Sciences, 2018
- A theoretical understanding of self-paced learningInformation Sciences, 2017