ReMem: Mutual Information-Aware Fine-tuning of Pretrained Vision Transformers for Effective Knowledge Distillation
Authors:
- Chengyu DongCDChengyu Dong
- Huan GuiHGHuan Gui
- Noveen SachdevaNSNoveen Sachdeva
- Long JinLJLong Jin
- Ke YinKYKe Yin
- Jingbo ShangJSJingbo Shang
- Lichan HongLHLichan Hong
- Ed H. ChiECEd H. Chi
- Zhe ZhaoZZZhe Zhao
Preprint
Abstract
Knowledge distillation from pretrained visual representation models offers an effective approach to improve small, task-specific production models. However, the effectiveness of such knowledge transfer drops significantly when distilling from strong models that are pretrained in a large scale. In this paper, we address this challenge for pretrained Vision Transformers (ViTs) by exploring methods to fine-tune them for more effective knowledge transfer. Motivated by the connection between mutual information and distillation effectiveness, we propose to employ mutual information-aware optimization during finetuning. For small or highly-imbalanced downstream datasets where such optimization becomes less effective, we introduce a simple yet effective heuristic of reweighting MLP blocks. This approach is inspired by our observation that top MLP blocks are primarily responsible for mutual information loss. Our method enables small student models to benefit from those pretrained models among the strongest.Keywords
All Related Versions
This publication has 4 references indexed in Scilit:
- Knowledge Distillation from a Stronger Teacher2022
- Efficient Knowledge Distillation from Model Checkpoints2022
- Training Deep Neural Networks in Generations: A More Tolerant Teacher Educates Better StudentsProceedings of the AAAI Conference on Artificial Intelligence, 2019
- Remote Sensing Image Scene Classification: Benchmark and State of the ArtProceedings of the IEEE, 2017