Lightweight Models for Classification and Detection via Knowledge Distillation and Distance Correlation
Open Access
- 17 December 2025
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1-9
Abstract
Knowledge distillation (KD) transfers what a large, accurate teacher model has learned to a smaller, faster student model. Standard losses such as Kullback–Leibler (KL) divergence, mean squared error (MSE), and dot product have been used successfully. Still, they often come with limitations, for example, assuming specific data distributions or requiring that teacher and student features have the same dimensionality. We introduce Distance Correlation (DC) as an additional distillation loss to get around these issues. DC measures non-linear dependencies between teacher and student representations and works without the need for matching dimensions, making it more flexible. We tested this idea on CIFAR-100 and Tiny ImageNet classification tasks, with Vision Transformers and ResNets as teachers and MobileNetV2 as the students. For object detection, we use a subset of MS COCO with YOLOv8-medium as the teacher and YOLOv8n as the student. Across these settings, combining KL divergence with DC consistently improves results up to a +5.6% boost in classification accuracy and +1.4% mAP in detection over the baseline. These gains suggest that DC is a practical, general purpose addition to existing KD objectives.Funding Information
- ANRF (SERB), Govt. of India (CRG/2020/006049)
This publication has 18 references indexed in Scilit:
- Masked Generative Distillation2022
- On the Versatile Uses of Partial Distance Correlation in Deep Learning2022
- Focal and Global Knowledge Distillation for Detectors2022
- TIDE: A General Toolbox for Identifying Object Detection Errors2020
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision ApplicationsarXiv, 2017
- You Only Look Once: Unified, Real-Time Object DetectionarXiv, 2016
- Deep Residual Learning for Image RecognitionarXiv, 2015
- Distilling the Knowledge in a Neural NetworkarXiv, 2015
- Measuring and testing dependence by correlation of distancesThe Annals of Statistics, 2007
- A vector space model for automatic indexingCommunications of the ACM, 1975