WebQA: Multihop and Multimodal QA
Authors:
Top Cited Papers
- 1 June 2022
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 16474-16483
Abstract
Scaling Visual Question Answering (VQA) to the open-domain and multi-hop nature of web searches, requires fundamental advances in visual representation learning, knowledge aggregation, and language generation. In this work, we introduce WEBQA, a challenging new benchmark that proves difficult for large-scale state-of-the-art models which lack language groundable visual representations for novel objects and the ability to reason, yet trivial for humans. WebQA mirrors the way humans use the web: 1) Ask a question, 2) Choose sources to aggregate, and 3) Produce a fluent language response. This is the behavior we should be expecting from IoT devices and digital assistants. Existing work prefers to assume that a model can either reason about knowledge in images or in text. WebQA includes a secondary text-only QA task to ensure improved visual performance does not come at the cost of language understanding. Our challenge for the community is to create unified multimodal reasoning models that answer questions regardless of the source modality, moving us closer to digital assistants that not only query language knowledge, but also the richer visual online world.Keywords
All Related Versions
This publication has 21 references indexed in Scilit:
- Constructing Datasets for Multi-hop Reading Comprehension Across DocumentsTransactions of the Association for Computational Linguistics, 2018
- HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering2018
- TVQA: Localized, Compositional Video Question Answering2018
- The Web as a Knowledge-Base for Answering Complex Questions2018
- Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering2017
- Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image AnnotationsInternational Journal of Computer Vision, 2017
- MovieQA: Understanding Stories in Movies through Question-Answering2016
- VQA: Visual Question Answering2015
- The Probabilistic Relevance Framework: BM25 and BeyondFoundations and Trends® in Information Retrieval, 2009
- BLEU2001