Do Better ImageNet Models Transfer Better?
Preprint
Abstract
Transfer learning is a cornerstone of computer vision, yet little work has been done to evaluate the relationship between architecture and transfer. An implicit hypothesis in modern computer vision research is that models that perform better on ImageNet necessarily perform better on other vision tasks. However, this hypothesis has never been systematically tested. Here, we compare the performance of 16 classification networks on 12 image classification datasets. We find that, when networks are used as fixed feature extractors or fine-tuned, there is a strong correlation between ImageNet accuracy and transfer accuracy ($r = 0.99$ and $0.96$, respectively). In the former setting, we find that this relationship is very sensitive to the way in which networks are trained on ImageNet; many common forms of regularization slightly improve ImageNet accuracy but yield penultimate layer features that are much worse for transfer learning. Additionally, we find that, on two small fine-grained image classification datasets, pretraining on ImageNet provides minimal benefits, indicating the learned features from ImageNet do not transfer well to fine-grained tasks. Together, our results show that ImageNet architectures generalize well across datasets, but ImageNet features are less general than previously suggested.
Keywords
All Related Versions
This publication has 13 references indexed in Scilit:
- Exploring the Limits of Weakly Supervised Pretraining2018
- Deep Layer Aggregation2018
- Mask-CNN: Localizing parts and selecting descriptors for fine-grained bird species categorizationPattern Recognition, 2018
- Object-Part Attention Model for Fine-Grained Image ClassificationIEEE Transactions on Image Processing, 2017
- Places: A 10 Million Image Database for Scene RecognitionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2017
- DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2017
- Coarse-to-Fine Description for Fine-Grained Visual CategorizationIEEE Transactions on Image Processing, 2016
- Factors of Transferability for a Generic ConvNet RepresentationIEEE Transactions on Pattern Analysis and Machine Intelligence, 2015
- The Pascal Visual Object Classes (VOC) ChallengeInternational Journal of Computer Vision, 2009
- On the limited memory BFGS method for large scale optimizationMathematical Programming, 1989