Generalized Kullback-Leibler Divergence Loss
Authors:
- Jiequan CuiJCJiequan Cui
- Beier ZhuBZBeier Zhu
- Qingshan XuQXQingshan Xu
- Zhuotao TianZTZhuotao Tian
- Xiaojuan QiXQXiaojuan Qi
- Bei YuBYBei Yu
- Hanwang ZhangHZHanwang Zhang
- Richang HongRHRichang Hong
Preprint
Abstract
In this paper, we delve deeper into the Kullback-Leibler (KL) Divergence loss and mathematically prove that it is equivalent to the Decoupled Kullback-Leibler (DKL) Divergence loss that consists of (1) a weighted Mean Square Error (wMSE) loss and (2) a Cross-Entropy loss incorporating soft labels. Thanks to the decoupled structure of DKL loss, we have identified two areas for improvement. Firstly, we address the limitation of KL loss in scenarios like knowledge distillation by breaking its asymmetric optimization property along with a smoother weight function. This modification effectively alleviates convergence challenges in optimization, particularly for classes with high predicted scores in soft labels. Secondly, we introduce class-wise global information into KL/DKL to reduce bias arising from individual samples. With these two enhancements, we derive the Generalized Kullback-Leibler (GKL) Divergence loss and evaluate its effectiveness by conducting experiments on CIFAR-10/100, ImageNet, and vision-language datasets, focusing on adversarial training, and knowledge distillation tasks. Specifically, we achieve new state-of-the-art adversarial robustness on the public leaderboard -- RobustBench and competitive knowledge distillation performance across CIFAR/ImageNet models and CLIP models, demonstrating the substantial practical merits. Our code is available at https://github.com/jiequancui/DKL.Keywords
All Related Versions
This publication has 37 references indexed in Scilit:
- Momentum Contrast for Unsupervised Visual Representation Learning2020
- Snapshot Distillation: Teacher-Student Optimization in One Generation2019
- MnasNet: Platform-Aware Neural Architecture Search for Mobile2019
- AutoAugment: Learning Augmentation Strategies From Data2019
- Class-Balanced Loss Based on Effective Number of Samples2019
- Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence2018
- A Gift from Knowledge Distillation: Fast Optimization, Network Minimization and Transfer Learning2017
- Perceptual Losses for Real-Time Style Transfer and Super-Resolution2016
- A Discriminative Feature Learning Approach for Deep Face Recognition2016
- Deep Residual Learning for Image Recognition2016