Cross-modality Multiple Relations Learning for Knowledge-based Visual Question Answering
Open Access
- 23 October 2023
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Multimedia Computing, Communications, and Applications
- Vol. 20 (3) , 1-22
Abstract
Knowledge-based visual question answering not only needs to answer the questions based on images but also incorporates external knowledge to study reasoning in the joint space of vision and language. To bridge the gap between visual content and semantic cues, it is important to capture the question-related and semantics-rich vision-language connections. Most existing solutions model simple intra-modality relation or represent cross-modality relation using a single vector, which makes it difficult to effectively model complex connections between visual features and question features. Thus, we propose a cross-modality multiple relations learning model, aiming to better enrich cross-modality representations and construct advanced multi-modality knowledge triplets. First, we design a simple yet effective method to generate multiple relations that represent the rich cross-modality relations. The various cross-modality relations link the textual question to the related visual objects. These multi-modality triplets efficiently align the visual objects and corresponding textual answers. Second, to encourage multiple relations to better align with different semantic relations, we further formulate a novel global-local loss. The global loss enables the visual objects and corresponding textual answers close to each other through cross-modality relations in the vision-language space, and the local loss better preserves semantic diversity among multiple relations. Experimental results on the Outside Knowledge VQA and Knowledge-Routed Visual Question Reasoning datasets demonstrate that our model outperforms the state-of-the-art methods.Keywords
Funding Information
- National Natural Science Foundation of China (62072212, 61906187, 61976207, and 61902394)
- Development Project of Jilin Province of China (20220508125RC, 20230201065GX, 20200401083GX, 2020C003, and 20200403172SF)
- National Key R&D Program (2018YFC2001302)
- Jilin Provincial Key Laboratory of Big Data Intelligent Cognition (20210504003GH)
This publication has 40 references indexed in Scilit:
- A Strong Baseline and Batch Normalization Neck for Deep Person Re-IdentificationIEEE Transactions on Multimedia, 2019
- Webly Supervised Joint Embedding for Cross-Modal Image-Text Retrieval2018
- Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering2018
- Beyond Bilinear: Generalized Multimodal Factorized High-Order Pooling for Visual Question AnsweringIEEE Transactions on Neural Networks and Learning Systems, 2018
- A Multimodal Translation-Based Approach for Knowledge Graph Representation Learning2018
- MUTAN: Multimodal Tucker Fusion for Visual Question Answering2017
- Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering2017
- Weakly-Shared Deep Transfer Networks for Heterogeneous-Domain Knowledge Propagation2015
- Microsoft COCO: Common Objects in Context2014
- ConceptNet — A Practical Commonsense Reasoning Tool-KitBT Technology Journal, 2004