Scene Graph Refinement Network for Visual Question Answering
- 22 April 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 25 (15209210) , 3950-3961
Abstract
Visual Question Answering aims to answer the free-form natural language question based on the visual clues in a given image. It is a difficult problem as it requires understanding the fine-grained structured information of both language and image for compositional reasoning. To establish the compositional reasoning, recent works attempt to introduce the scene graph in VQA. However, as the generated scene graphs are usually quite noisy, it greatly limits the performance of question answering. Therefore, this paper proposes to refine the scene graphs for improving the effectiveness. Specifically, we present a novel Scene Graph Refinement network (SGR), which introduces a transformer-based refinement network to enhance the object and relation features for better classification. Moreover, as the question provides valuable clues for distinguishing whether the subject, predicate, objecttriplets are helpful or not, the SGR network exploits the semantic information presented in the questions to select the most relevant relations for question answering. Extensive experiments conducted on the GQA benchmark demonstrate the effectiveness of our method.Keywords
Funding Information
- National Natural Science Foundation of China (62072116)
- Shanghai Science and Technology Program Project (21JC1400600)
- Shuguang Program
- Shanghai Education Development Foundation
- Shanghai Municipal Education Commission
This publication has 59 references indexed in Scilit:
- Multi-level Attention Networks for Visual Question Answering2017
- Graph-Structured Representations for Visual Question Answering2017
- Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image AnnotationsInternational Journal of Computer Vision, 2017
- Deep-based Ingredient Recognition for Cooking Recipe Retrieval2016
- Visual Relationship Detection with Language Priors2016
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal NetworksIEEE Transactions on Pattern Analysis and Machine Intelligence, 2016
- Stacked Attention Networks for Image Question Answering2016
- Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding2016
- VQA: Visual Question Answering2015
- Glove: Global Vectors for Word Representation2014