Few-Shot Learning of Compact Models via Task-Specific Meta Distillation
Authors:
- 1 January 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 6254-6263
Abstract
We consider a new problem of few-shot learning of com-pact models. Meta-learning is a popular approach for few-shot learning. Previous work in meta-learning typically assumes that the model architecture during meta-training is the same as the model architecture used for final deployment. In this paper, we challenge this basic assumption. For final deployment, we often need the model to be small. But small models usually do not have enough capacity to effectively adapt to new tasks. In the mean time, we often have access to the large dataset and extensive computing power during meta-training since meta-training is typically per-formed on a server. In this paper, we propose task-specific meta distillation that simultaneously learns two models in meta-learning: a large teacher model and a small student model. These two models are jointly learned during meta-training. Given a new task during meta-testing, the teacher model is first adapted to this task, then the adapted teacher model is used to guide the adaptation of the student model. The adapted student model is used for final deployment. We demonstrate the effectiveness of our approach in few-shot image classification using model-agnostic meta-learning (MAML). Our proposed method outperforms other alternatives on several benchmark datasets.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China
- China Scholarship Council
This publication has 29 references indexed in Scilit:
- Probabilistic Knowledge Transfer for Lightweight Deep Representation LearningIEEE Transactions on Neural Networks and Learning Systems, 2020
- Hierarchical Knowledge Squeezed Adversarial Network CompressionProceedings of the AAAI Conference on Artificial Intelligence, 2020
- Similarity-Preserving Knowledge Distillation2019
- CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features2019
- Variational Information Distillation for Knowledge Transfer2019
- Knowledge Distillation via Instance Relationship Graph2019
- Relational Knowledge Distillation2019
- Learning to Compare: Relation Network for Few-Shot Learning2018
- Deep Residual Learning for Image Recognition2016
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015