Visual Madlibs: Fill in the Blank Description Generation and Question Answering
Authors:
- Licheng YuLYLicheng Yu
- Eunbyung ParkEPEunbyung Park
- Alexander C. BergABAlexander C. Berg
- Tamara L. BergTBTamara L. Berg
- 1 December 2015
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 2461-2469
Abstract
In this paper, we introduce a new dataset consisting of 360,001 focused natural language descriptions for 10,738 images. This dataset, the Visual Madlibs dataset, is collected using automatically produced fill-in-the-blank templates designed to gather targeted descriptions about: people and objects, their appearances, activities, and interactions, as well as inferences about the general scene or its broader context. We provide several analyses of the Visual Madlibs dataset and demonstrate its applicability to two new description generation tasks: focused description generation, and multiple-choice question-answering for images. Experiments using joint-embedding and deep learning methods show promising results on these tasks.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- Visual Turing test for computer vision systemsProceedings of the National Academy of Sciences, 2015
- Translating Videos to Natural Language Using Deep Recurrent Neural Networks2015
- From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptionsTransactions of the Association for Computational Linguistics, 2014
- Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation2014
- A Fast and Accurate Dependency Parser using Neural Networks2014
- Question Answering with Subgraph Embeddings2014
- Open Question Answering with Weakly Supervised Embedding Models2014
- A Multi-View Embedding Space for Modeling Internet Images, Tags, and Their SemanticsInternational Journal of Computer Vision, 2013
- Understanding and predicting importance in images2012
- Baby talk: Understanding and generating simple image descriptions2011