Abstract
Motivated by the limitation that most contrastive knowledge distillation methods optimize only one perspective, this paper proposes a simple and efficient contrastive knowledge distillation method that formulates distillation as a dual-perspective objective over the class-wise and the sample-wise. Unlike prior contrastive knowledge distillation, we jointly align teacher and student logit distributions at both perspectives. Concretely, same input pairs serve as anchors whose similarities are optimized against negatives drawn from other classes, enforcing class-wise alignment and maximizing separability between classes. To counterbalance the potential distortion of sample geometry caused by pure class-wise alignment, we simultaneously perform sample-wise alignment, matching each student logit only to its teacher counterpart for the same input, irrespective of class relations. This dual alignment supplies complementary supervision, stabilizes sample-wise structure, and sharpens decision boundaries. Moreover, we add a label-guided same class contrast that promotes compactness within classes while further enhancing separability between classes. We further propose a delayed learning strategy that orchestrates multiple contrastive losses, enabling a more effective integration that preserves their respective advantages. Across CIFAR-100, ImageNet, Tiny-ImageNet, and STL-10, our approach surpasses most logit-based and feature-based baselines in image classification, transfer learning, and ViT distillation, while maintaining superior training efficiency.

This publication has 9 references indexed in Scilit: