Rethinking the Hyperparameters for Fine-tuning
Preprint
Abstract
Fine-tuning from pre-trained ImageNet models has become the de-facto standard for various computer vision tasks. Current practices for fine-tuning typically involve selecting an ad-hoc choice of hyperparameters and keeping them fixed to values normally used for training from scratch. This paper re-examines several common practices of setting hyperparameters for fine-tuning. Our findings are based on extensive empirical evaluation for fine-tuning on various transfer learning benchmarks. (1) While prior works have thoroughly investigated learning rate and batch size, momentum for fine-tuning is a relatively unexplored parameter. We find that the value of momentum also affects fine-tuning performance and connect it with previous theoretical findings. (2) Optimal hyperparameters for fine-tuning, in particular, the effective learning rate, are not only dataset dependent but also sensitive to the similarity between the source domain and target domain. This is in contrast to hyperparameters for training from scratch. (3) Reference-based regularization that keeps models close to the initial model does not necessarily apply for "dissimilar" datasets. Our findings challenge common practices of fine-tuning and encourages deep learning practitioners to rethink the hyperparameters for fine-tuning.Keywords
All Related Versions
This publication has 19 references indexed in Scilit:
- Bag of Tricks for Image Classification with Convolutional Neural Networks2019
- Exploring the Limits of Weakly Supervised Pretraining2018
- Large Scale Fine-Grained Categorization and Domain-Specific Transfer Learning2018
- The iNaturalist Species Classification and Detection Dataset2018
- Borrowing Treasures from the Wealthy: Deep Transfer Learning through Selective Joint Fine-Tuning2017
- Identity Mappings in Deep Residual Networks2016
- Deep Residual Learning for Image Recognition2016
- Going deeper with convolutions2015
- Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation2014
- What Does Classifying More Than 10,000 Image Categories Tell Us?2010