Reciprocal Teacher-Student Learning via Forward and Feedback Knowledge Distillation
Top Cited Papers
- 14 March 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 26 (15209210) , 7901-7916
Abstract
Knowledge distillation (KD) is a prevalent model compression technique in deep learning, aiming to leverage knowledge from a large teacher model to enhance the training of a smaller student model. It has found success in deploying compact deep models in intelligent applications like intelligent transportation, smart health, and distributed intelligence. Current knowledge distillation methods primarily fall into two categories: offline and online knowledge distillation. Offline methods involve a one-way distillation process, transferring unvaried knowledge from teacher to student, while online methods enable the simultaneous training of multiple peer students. However, existing knowledge distillation methods often face challenges where the student may not fully comprehend the teacher's knowledge due to model capacity gaps, and there might be knowledge incongruence among outputs of multiple students without teacher guidance. To address these issues, we propose a novel reciprocal teacher-student learning inspired by human teaching and examining through forward and feedback knowledge distillation (FFKD). Forward knowledge distillation operates offline, while feedback knowledge distillation follows an online scheme. The rationale is that feedback knowledge distillation enables the pre-trained teacher model to receive feedback from students, allowing the teacher to refine its teaching strategies accordingly. To achieve this, we introduce a new weighting constraint to gauge the extent of students' understanding of the teacher's knowledge, which is then utilized to enhance teaching strategies. Experimental results on five visual recognition datasets demonstrate that the proposed FFKD outperforms current state-of-the-art knowledge distillation methods.Keywords
Funding Information
- National Natural Science Foundation of China (61976107, 61502208)
This publication has 51 references indexed in Scilit:
- Multi-target Knowledge Distillation via Student Self-reflectionInternational Journal of Computer Vision, 2023
- Teacher or supervisor? Effective online knowledge distillation via guided collaborative learningComputer Vision and Image Understanding, 2023
- Task-balanced distillation for object detectionPattern Recognition, 2023
- Diversified branch fusion for self-knowledge distillationInformation Fusion, 2022
- Real-time and light-weighted unsupervised video object segmentation networkPattern Recognition, 2021
- Knowledge Distillation via Channel Correlation Structure2021
- Knowledge Distillation: A SurveyInternational Journal of Computer Vision, 2021
- Knowledge Transfer via Dense Cross-Layer Mutual-Distillation2020
- ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design2018
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015