DCKD: Dual-perspective Contrastive Knowledge Distillation
Preprint
- 21 April 2026
- preprint
- Published by Elsevier in SSRN Electronic Journal
Abstract
Motivated by the limitation that most contrastive knowledge distillation methods optimize only one perspective, this paper proposes a simple and efficient contrastive knowledge distillation method that formulates distillation as a dual-perspective objective over the class-wise and the sample-wise. Unlike prior contrastive knowledge distillation, we jointly align teacher and student logit distributions at both perspectives. Concretely, same input pairs serve as anchors whose similarities are optimized against negatives drawn from other classes, enforcing class-wise alignment and maximizing separability between classes. To counterbalance the potential distortion of sample geometry caused by pure class-wise alignment, we simultaneously perform sample-wise alignment, matching each student logit only to its teacher counterpart for the same input, irrespective of class relations. This dual alignment supplies complementary supervision, stabilizes sample-wise structure, and sharpens decision boundaries. Moreover, we add a label-guided same class contrast that promotes compactness within classes while further enhancing separability between classes. We further propose a delayed learning strategy that orchestrates multiple contrastive losses, enabling a more effective integration that preserves their respective advantages. Across CIFAR-100, ImageNet, Tiny-ImageNet, and STL-10, our approach surpasses most logit-based and feature-based baselines in image classification, transfer learning, and ViT distillation, while maintaining superior training efficiency.Keywords
This publication has 9 references indexed in Scilit:
- Adaptive Knowledge Distillation With Attention-Based Multi-Modal Fusion for Robust Dim Object DetectionIEEE Transactions on Multimedia, 2024
- Curriculum Temperature for Knowledge DistillationProceedings of the AAAI Conference on Artificial Intelligence, 2023
- Dynamic Contrastive Distillation for Image-Text RetrievalIEEE Transactions on Multimedia, 2023
- Knowledge Distillation from a Stronger Teacher2022
- Improved Knowledge Distillation via Teacher AssistantProceedings of the AAAI Conference on Artificial Intelligence, 2020
- CKD: Cross-Task Knowledge Distillation for Text-to-Image SynthesisIEEE Transactions on Multimedia, 2019
- Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden NeuronsProceedings of the AAAI Conference on Artificial Intelligence, 2019
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal NetworksIEEE Transactions on Pattern Analysis and Machine Intelligence, 2016
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015