CrowdVerge
Authors:
Open Access
- 2 May 2017
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 3511-3522
Abstract
Visual question answering systems empower users to ask any question about any image and receive a valid answer. However, existing systems do not yet account for the fact that a visual question can lead to a single answer or multiple different answers. While a crowd often agrees, disagreements do arise for many reasons including that visual questions are ambiguous, subjective, or difficult. We propose a model, CrowdVerge, for automatically predicting from a visual question whether a crowd would agree on one answer. We then propose how to exploit these predictions in a novel application to efficiently collect all valid answers to visual questions. Specifically, we solicit fewer human responses when answer agreement is expected and more human responses otherwise. Experiments on 121,811 visual questions asked by sighted and blind people show that, compared to existing crowdsourcing systems, our system captures the same answer diversity with typically 14-23% less crowd involvement.Keywords
Funding Information
- Office of Naval Research (YIP N00014-12-1-0754)
- National Science Foundation (IIS-1065390)
This publication has 19 references indexed in Scilit:
- Image specificity2015
- Salient Object Subitizing2015
- Zensors2015
- Gauging Receptiveness to Social Microvolunteering2015
- RegionSpeak2015
- Scalable multi-label annotation2014
- Answering visual questions with conversational crowd assistants2013
- Visual challenges in the everyday lives of blind people2013
- Cost-Sensitive Active Visual Category LearningInternational Journal of Computer Vision, 2010
- Labeling images with a computer game2004