A Unified Asymmetric Knowledge Distillation Framework for Image Classification
Open Access
- 10 May 2024
- journal article
- research article
- Published by Springer Nature in Neural Processing Letters
- Vol. 56 (3) , 1-25
Abstract
Knowledge distillation is a model compression technique that transfers knowledge learned by teacher networks to student networks. Existing knowledge distillation methods greatly expand the forms of knowledge, but also make the distillation models complex and symmetric. However, few studies have explored the commonalities among these methods. In this study, we propose a concise distillation framework to unify these methods and a method to construct asymmetric knowledge distillation under the framework. Asymmetric distillation aims to enable differentiated knowledge transfers for different distillation objects. We designed a multi-stage shallow-wide branch bifurcation method to distill different knowledge representations and a grouping ensemble strategy to supervise the network to teach and learn selectively. Consequently, we conducted experiments using image classification benchmarks to verify the proposed method. Experimental results show that our implementation can achieve considerable improvements over existing methods, demonstrating the effectiveness of the method and the potential of the framework.Keywords
Funding Information
- Zhejiang Provincial Natural Science Foundation of China (LDT23F01013F01, LDT23F01013F01, LDT23F01013F01, LDT23F01013F01, LDT23F01013F01)
- Fundamental Research Funds for the Central Universities
This publication has 42 references indexed in Scilit:
- Collaborative knowledge distillation via filter knowledge transferExpert Systems with Applications, 2023
- Mutual Information-Based Neural Network Distillation for Improving Photonic Neural Network TrainingNeural Processing Letters, 2023
- Knowledge Fusion Distillation: Improving Distillation with Multi-scale Attention MechanismsNeural Processing Letters, 2023
- COLAM: Co-Learning of Deep Neural Networks and Soft Labels via Alternating MinimizationNeural Processing Letters, 2022
- Online Ensemble Model Compression Using Knowledge Distillation2020
- Knowledge Distillation Meets Self-supervision2020
- Improving Knowledge Distillation via Category Structure2020
- Adaptive multi-teacher multi-level knowledge distillationNeurocomputing, 2020
- Self-supervised Knowledge Distillation Using Singular Value Decomposition2018
- ImageNet: A large-scale hierarchical image database2009