Divide and Conquer: Answering Questions with Object Factorization and Compositional Reasoning
Authors:
- 1 June 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 6736-6745
Abstract
Humans have the innate capability to answer diverse questions, which is rooted in the natural ability to correlate different concepts based on their semantic relationships and decompose difficult problems into sub-tasks. On the contrary, existing visual reasoning methods assume training samples that capture every possible object and reasoning problem, and rely on black-boxed models that commonly exploit statistical priors. They have yet to develop the capability to address novel objects or spurious biases in real-world scenarios, and also fall short of interpreting the rationales behind their decisions. Inspired by humans' reasoning of the visual world, we tackle the aforementioned challenges from a compositional perspective, and propose an integral framework consisting of a principled object factorization method and a novel neural module network. Our factorization method decomposes objects based on their key characteristics, and automatically derives prototypes that represent a wide range of objects. With these prototypes encoding important semantics, the proposed network then correlates objects by measuring their similarity on a common semantic space and makes decisions with a compositional reasoning process. It is capable of answering questions with diverse objects regard-less of their availability during training, and overcoming the issues of biased question-answer distributions. In addition to the enhanced generalizability, our framework also provides an interpretable interface for understanding the decision-making process of models. Our code is available at https://github.com/szzexpoi/POEM.Keywords
All Related Versions
Funding Information
- NSF (2143197,2227450)
This publication has 44 references indexed in Scilit:
- Learning Spatial Regularization with Image-Level Supervisions for Multi-label Image Classification2017
- Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image AnnotationsInternational Journal of Computer Vision, 2017
- Neural Module Networks2016
- Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding2016
- VQA: Visual Question Answering2015
- Visual Turing test for computer vision systemsProceedings of the National Academy of Sciences, 2015
- Glove: Global Vectors for Word Representation2014
- A Review on Multi-Label Learning AlgorithmsIEEE Transactions on Knowledge and Data Engineering, 2013
- DEDUCTIVE REASONINGAnnual Review of Psychology, 1999
- Least squares quantization in PCMIEEE Transactions on Information Theory, 1982