Knowledge Distillation via the Target-aware Transformer
Authors:
- 1 June 2022
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 10905-10914
Abstract
Knowledge distillation becomes a de facto standard to improve the performance of small neural networks. Most of the previous works propose to regress the representational features from the teacher to the student in a one-to-one spatial matching fashion. However, people tend to overlook the fact that, due to the architecture differences, the semantic information on the same spatial location usually vary. This greatly undermines the underlying assumption of the one-to-one distillation approach. To this end, we propose a novel one-to-all spatial matching knowledge distillation approach. Specifically, we allow each pixel of the teacher feature to be distilled to all spatial locations of the student features given its similarity, which is generated from a target-aware transformer. Our approach surpasses the state-of-the-art methods by a significant margin on various computer vision benchmarks, such as ImageNet, Pascal VOC and COCOStuff10k. Code is available at https://github.com/sihaoevery/TaT.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (NSFC) (61976233)
- National Key Research and Development Program of China (2020AAA0108104)
- Australian Research Council (ARC)
This publication has 30 references indexed in Scilit:
- Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation2018
- Learning without ForgettingIEEE Transactions on Pattern Analysis and Machine Intelligence, 2017
- Deformable Convolutional Networks2017
- A Gift from Knowledge Distillation: Fast Optimization, Network Minimization and Transfer Learning2017
- Deep Residual Learning for Image Recognition2016
- Going deeper with convolutions2015
- Microsoft COCO: Common Objects in Context2014
- Semantic contours from inverse detectors2011
- The Pascal Visual Object Classes (VOC) ChallengeInternational Journal of Computer Vision, 2009
- ImageNet: A large-scale hierarchical image database2009