Ask Your Neurons: A Neural-Based Approach to Answering Questions about Images
Authors:
- Mateusz MalinowskiMMMateusz Malinowski
- Marcus RohrbachMRMarcus Rohrbach
- Mario FritzMFMario Fritz
Top Cited Papers
- 1 December 2015
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-9
Abstract
We address a question answering task on real-world images that is set up as a Visual Turing Test. By combining latest advances in image representation and natural language processing, we propose Neural-Image-QA, an end-to-end formulation to this problem for which all parts are trained jointly. In contrast to previous efforts, we are facing a multi-modal problem where the language output (answer) is conditioned on visual and natural language input (image and question). Our approach Neural-Image-QA doubles the performance of the previous best approach on this problem. We provide additional insights into the problem by analyzing how much information is contained only in the language part for which we provide a new human baseline. To study human consensus, which is related to the ambiguities inherent in this challenging task, we propose two novel metrics and collect additional answers which extends the original DAQUAR dataset to DAQUAR-Consensus.Keywords
All Related Versions
This publication has 19 references indexed in Scilit:
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015
- Visual Turing test for computer vision systemsProceedings of the National Academy of Sciences, 2015
- Semantic Parsing via Paraphrasing2014
- Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation2014
- Learning Dependency-Based Compositional SemanticsComputational Linguistics, 2013
- Indoor Segmentation and Support Inference from RGBD Images2012
- Gradient-based learning applied to document recognitionProceedings of the IEEE, 1998
- Long Short-Term MemoryNeural Computation, 1997
- The Equivalence of Weighted Kappa and the Intraclass Correlation Coefficient as Measures of ReliabilityEducational and Psychological Measurement, 1973
- A Coefficient of Agreement for Nominal ScalesEducational and Psychological Measurement, 1960