Learning Metrics From Teachers: Compact Networks for Image Embedding
Authors:
- Lu YuLYLu Yu
- 1 June 2019
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 2902-2911
Abstract
Metric learning networks are used to compute image embeddings, which are widely used in many applications such as image retrieval and face recognition. In this paper, we propose to use network distillation to efficiently compute image embeddings with small networks. Network distillation has been successfully applied to improve image classification, but has hardly been explored for metric learning. To do so, we propose two new loss functions that model the communication of a deep teacher network to a small student network. We evaluate our system in several datasets, including CUB-200-2011, Cars-196, Stanford Online Products and show that embeddings computed using small student networks perform significantly better than those computed using standard networks of similar size. Results on a very compact network (MobileNet-0.25), which can be used on mobile devices, show that the proposed method can greatly improve Recall@1 results from 27.5\% to 44.6\%. Furthermore, we investigate various aspects of distillation for embeddings, including hint and attention layers, semi-supervised learning and cross quality distillation. (Code is available at https://github.com/yulu0724/EmbeddingDistillation).Keywords
All Related Versions
This publication has 15 references indexed in Scilit:
- Discriminative Learning of Deep Convolutional Feature Point Descriptors2015
- Unsupervised Learning of Visual Representations Using Videos2015
- Deep Metric Learning Using Triplet Network2015
- FaceNet: A unified embedding for face recognition and clustering2015
- Learning Fine-Grained Image Similarity with Deep Ranking2014
- CNN Features Off-the-Shelf: An Astounding Baseline for Recognition2014
- 3D Object Representations for Fine-Grained Categorization2013
- Metric Learning: A SurveyFoundations and Trends in Machine Learning, 2013
- Model compression2006
- Learning a Similarity Metric Discriminatively, with Application to Face Verification2005