Measuring Machine Intelligence Through Visual Question Answering
Preprint
Abstract
As machines have become more intelligent, there has been a renewed interest in methods for measuring their intelligence. A common approach is to propose tasks for which a human excels, but one which machines find difficult. However, an ideal task should also be easy to evaluate and not be easily gameable. We begin with a case study exploring the recently popular task of image captioning and its limitations as a task for measuring machine intelligence. An alternative and more promising task is Visual Question Answering that tests a machine's ability to reason about language and vision. We describe a dataset unprecedented in size created for the task that contains over 760,000 human generated questions about images. Using around 10 million human generated answers, machines may be easily evaluated.Keywords
All Related Versions
This publication has 6 references indexed in Scilit:
- Visual Turing test for computer vision systemsProceedings of the National Academy of Sciences, 2015
- Adopting Abstract Images for Semantic Scene UnderstandingIEEE Transactions on Pattern Analysis and Machine Intelligence, 2014
- Joint Video and Text Parsing for Understanding Events and Answering QueriesIEEE MultiMedia, 2014
- Framing Image Description as a Ranking Task: Data, Models and Evaluation MetricsJournal of Artificial Intelligence Research, 2013
- BabyTalk: Understanding and Generating Simple Image DescriptionsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2013
- MCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text2013