Representation Similarity Analysis for Efficient Task Taxonomy & Transfer Learning
Authors:
- 1 June 2019
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 12379-12388
Abstract
Transfer learning is widely used in deep neural network models when there are few labeled examples available. The common approach is to take a pre-trained network in a similar task and finetune the model parameters. This is usually done blindly without a pre-selection from a set of pre-trained models, or by finetuning a set of models trained on different tasks and selecting the best performing one by cross-validation. We address this problem by proposing an approach to assess the relationship between visual tasks and their task-specific models. Our method uses Representation Similarity Analysis (RSA), which is commonly used to find a correlation between neuronal responses from brain data and models. With RSA we obtain a similarity score among tasks by computing correlations between models trained on different tasks. Our method is efficient as it requires only pre-trained models, and a few images with no further training. We demonstrate the effectiveness and efficiency of our method to generating task taxonomy on Taskonomy dataset. We next evaluate the relationship of RSA with the transfer learning performance on Taskonomy tasks and a new task: Pascal VOC semantic segmentation. Our results reveal that models trained on tasks with higher similarity score show higher transfer learning performance. Surprisingly, the best transfer learning result for Pascal VOC semantic segmentation is not obtained from the pre-trained model on semantic segmentation, probably due to the domain differences, and our method successfully selects the high performing models.Keywords
All Related Versions
This publication has 22 references indexed in Scilit:
- Comparison of deep neural networks to spatio-temporal cortical dynamics of human visual object recognition reveals hierarchical correspondenceScientific Reports, 2016
- Deep Residual Learning for Image Recognition2016
- Deep convolutional neural fields for depth estimation from a single image2015
- Fully convolutional networks for semantic segmentation2015
- Deep Supervised, but Not Unsupervised, Models May Explain IT Cortical RepresentationPLOS Computational Biology, 2014
- Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation2014
- A Toolbox for Representational Similarity AnalysisPLOS Computational Biology, 2014
- Resolving human object recognition in space and timeNature Neuroscience, 2014
- The Pascal Visual Object Classes (VOC) ChallengeInternational Journal of Computer Vision, 2009
- Representational similarity analysis – connecting the branches of systems neuroscienceFrontiers in Systems Neuroscience, 2008