Relation-Aware Heterogeneous Graph Network for Learning Intermodal Semantics in Textbook Question Answering
- 23 April 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks and Learning Systems
- Vol. 35 (9) , 11872-11883
Abstract
Textbook question answering (TQA) task aims to infer answers for given questions from a multimodal context, including text and diagrams. The existing studies have aggregated intramodal semantics extracted from a single modality but have yet to capture the intermodal semantics between different modalities. A major challenge in learning intermodal semantics is maintaining lossless intramodal semantics while bridging the gap of semantics caused by heterogeneity. In this article, we propose an intermodal relation-aware heterogeneous graph network (IMR-HGN) to extract the intermodal semantics for TQA, which aggregates different modalities while learning features rather than representing them independently. First, we design a multidomain consistent representation (MDCR) to eliminate semantic gaps by capturing intermodal features while maintaining lossless intramodal semantics in multidomains. Furthermore, we present neighbor-based relation inpainting (NRI) to reduce semantic ambiguity via repairing fuzzy relations with correlations of relations. Finally, we propose hierarchical multisemantics aggregation (HMSA) to guarantee the completeness of semantics by aggregating features of nodes and relations with a reconstruction network (RN). Experimental results show that IMR-HGN could extract the intermodal semantics of answers, achieving a 2.16% improvement on the validation set of the TQA dataset and a 3.04% increase on the test set of the AI2D dataset.Keywords
Funding Information
- Project of Science and Technology Research and Development Plan of China Railway Corporation (N2023J044)
This publication has 43 references indexed in Scilit:
- Centrality-based Relation aware Heterogeneous Graph Neural NetworkKnowledge-Based Systems, 2023
- Multiscale Feature Extraction and Fusion of Image and Text in VQAInternational Journal of Computational Intelligence Systems, 2023
- Dynamic dual graph networks for textbook question answeringPattern Recognition, 2023
- Beyond OCR + VQA: Towards end-to-end reading and reasoning for robust and accurate textvqaPattern Recognition, 2023
- FAST-VQA: Efficient End-to-End Video Quality Assessment with Fragment Sampling2022
- Knowledge base graph embedding module design for Visual question answering modelPattern Recognition, 2021
- Textbook Question Answering with Multi-type Question Learning and Contextualized Diagram Representation2021
- MoQA – A Multi-modal Question Answering Architecture2019
- A Diagram is Worth a Dozen Images2016
- Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering2016