Predicting Deep Zero-Shot Convolutional Neural Networks Using Textual Descriptions
Authors:
- Jimmy Lei BaJBJimmy Lei Ba
- Kevin SwerskyKSKevin Swersky
- Sanja FidlerSFSanja Fidler
- Ruslan SalakhutdinovRSRuslan Salakhutdinov
Top Cited Papers
- 1 December 2015
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 4247-4255
Abstract
One of the main challenges in Zero-Shot Learning of visual categories is gathering semantic attributes to accompany images. Recent work has shown that learning from textual descriptions, such as Wikipedia articles, avoids the problem of having to explicitly define these attributes. We present a new model that can classify unseen categories from their textual description. Specifically, we use text features to predict the output weights of both the convolutional and the fully connected layers in a deep convolutional neural network (CNN). We take advantage of the architecture of CNNs and learn features at different layers, rather than just learning an embedding space for both modalities, as is common with existing approaches. The proposed model also allows us to automatically generate a list of pseudo-attributes for each visual category consisting of words from Wikipedia articles. We train our models end-to-end using the Caltech-UCSD bird and flower datasets and evaluate both ROC and Precision-Recall curves. Our empirical results show that the proposed model significantly outperforms previous methods.Keywords
All Related Versions
This publication has 13 references indexed in Scilit:
- Write a Classifier: Zero-Shot Learning Using Purely Textual Descriptions2013
- Undoing the Damage of Dataset Bias2012
- Domain adaptation for object recognition: An unsupervised approach2011
- What you saw is not what you get: Domain adaptation using asymmetric kernel transforms2011
- Unbiased look at dataset bias2011
- Describing objects by their attributes2009
- Learning to detect unseen object classes by between-class attribute transfer2009
- Automated Flower Classification over a Large Number of Classes2008
- A Bayesian approach to unsupervised one-shot learning of object categories2003
- Long Short-Term MemoryNeural Computation, 1997