Aligned Image-Word Representations Improve Inductive Transfer Across Vision-Language Tasks
Preprint
Abstract
A grand goal of computer vision is to build systems that learn visual representations over time that can be applied to many tasks. In this paper, we investigate a vision-language embedding as a core representation and show that it leads to better cross-task transfer than standard multi-task learning. In particular, the task of visual recognition is aligned to the task of visual question answering by forcing each to use the same word-region embeddings. We show this leads to greater inductive transfer from recognition to VQA than standard multitask learning. Visual recognition also improves, especially for categories that have relatively few recognition training labels but appear often in the VQA setting. Thus, our paper takes a small step towards creating more general vision systems by showing the benefit of interpretable, flexible, and trainable core representations.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- Attention Correctness in Neural Image CaptioningProceedings of the AAAI Conference on Artificial Intelligence, 2017
- Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image AnnotationsInternational Journal of Computer Vision, 2017
- Deep Residual Learning for Image Recognition2016
- Stacked Attention Networks for Image Question Answering2016
- DenseCap: Fully Convolutional Localization Networks for Dense Captioning2016
- Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding2016
- Learning to Compose Neural Networks for Question Answering2016
- Framing Image Description as a Ranking Task: Data, Models and Evaluation MetricsJournal of Artificial Intelligence Research, 2013
- Representation Learning: A Review and New PerspectivesIEEE Transactions on Pattern Analysis and Machine Intelligence, 2013
- Toward an Architecture for Never-Ending Language LearningProceedings of the AAAI Conference on Artificial Intelligence, 2010