Aligned Image-Word Representations Improve Inductive Transfer Across Vision-Language Tasks
Authors:
- Tanmay GuptaTGTanmay Gupta
- Kevin ShihKSKevin Shih
- Saurabh SinghSSSaurabh Singh
- Derek HoiemDHDerek Hoiem
- 1 October 2017
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 4223-4232
Abstract
An important goal of computer vision is to build systems that learn visual representations over time that can be applied to many tasks. In this paper, we investigate a vision-language embedding as a core representation and show that it leads to better cross-task transfer than standard multitask learning. In particular, the task of visual recognition is aligned to the task of visual question answering by forcing each to use the same word-region embeddings. We show this leads to greater inductive transfer from recognition to VQA than standard multitask learning. Visual recognition also improves, especially for categories that have relatively few recognition training labels but appear often in the VQA setting. Thus, our paper takes a small step towards creating more general vision systems by showing the benefit of interpretable, flexible, and trainable core representations.Keywords
All Related Versions
This publication has 23 references indexed in Scilit:
- Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models2015
- VQA: Visual Question Answering2015
- Curriculum learning of multiple tasks2015
- From captions to visual concepts and back2015
- Improving Image-Sentence Embeddings Using Large Weakly Annotated Photo Collections2014
- NEIL: Extracting Visual Knowledge from Web Data2013
- Representation Learning: A Review and New PerspectivesIEEE Transactions on Pattern Analysis and Machine Intelligence, 2013
- Lifelong Learning Algorithms1998
- Multitask Learning1998
- WordNetCommunications of the ACM, 1995