Decoupled Kullback-Leibler Divergence Loss
Authors:
- Zhuotao TianZTZhuotao Tian
- Xiaojuan QiXQXiaojuan Qi
- Hanwang ZhangHZHanwang Zhang
Preprint
Abstract
In this paper, we delve deeper into the Kullback-Leibler (KL) Divergence loss and observe that it is equivalent to the Doupled Kullback-Leibler (DKL) Divergence loss that consists of 1) a weighted Mean Square Error (wMSE) loss and 2) a Cross-Entropy loss incorporating soft labels. From our analysis of the DKL loss, we have identified two areas for improvement. Firstly, we address the limitation of DKL in scenarios like knowledge distillation by breaking its asymmetry property in training optimization. This modification ensures that the wMSE component is always effective during training, providing extra constructive cues. Secondly, we introduce global information into DKL for intra-class consistency regularization. With these two enhancements, we derive the Improved Kullback-Leibler (IKL) Divergence loss and evaluate its effectiveness by conducting experiments on CIFAR-10/100 and ImageNet datasets, focusing on adversarial training and knowledge distillation tasks. The proposed approach achieves new state-of-the-art performance on both tasks, demonstrating the substantial practical merits. Code and models will be available soon at https://github.com/jiequancui/DKL.Keywords
All Related Versions
This publication has 24 references indexed in Scilit:
- Momentum Contrast for Unsupervised Visual Representation Learning2020
- Equalization Loss for Long-Tailed Object Recognition2020
- Large-Scale Long-Tailed Recognition in an Open World2019
- Snapshot Distillation: Teacher-Student Optimization in One Generation2019
- MnasNet: Platform-Aware Neural Architecture Search for Mobile2019
- Relational Knowledge Distillation2019
- A Gift from Knowledge Distillation: Fast Optimization, Network Minimization and Transfer Learning2017
- Perceptual Losses for Real-Time Style Transfer and Super-Resolution2016
- A Discriminative Feature Learning Approach for Deep Face Recognition2016
- Deep Residual Learning for Image Recognition2016