MMKGR: Multi-hop Multi-modal Knowledge Graph Reasoning
Authors:
- 1 April 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 96-109
Abstract
Multi-modal knowledge graphs (MKGs) include not only the relation triplets, but also related multi-modal auxiliary data (i.e., texts and images), which enhance the diversity of knowledge. However, the natural incompleteness has significantly hindered the applications of MKGs. To tackle the problem, existing studies employ the embedding-based reasoning models to infer the missing knowledge after fusing the multi-modal features. However, the reasoning performance of these methods is limited due to the following problems: (1) ineffective fusion of multi-modal auxiliary features; (2) lack of complex reasoning ability as well as inability to conduct the multi-hop reasoning which is able to infer more missing knowledge. To overcome these problems, we propose a novel model entitled MMKGR (Multi-hop Multi-modal Knowledge Graph Reasoning). Specifically, the model contains the following two components: (1) a unified gate-attention network which is designed to generate effective multi-modal complementary features through sufficient attention interaction and noise reduction; (2) a complementary feature-aware reinforcement learning method which is proposed to predict missing elements by performing the multi-hop reasoning process, based on the features obtained in component (1). The experimental results demonstrate that MMKGR outperforms the state-of-the-art approaches in the MKG reasoning task.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China
- Australian Research Council
This publication has 60 references indexed in Scilit:
- IMGpedia: A Linked Dataset with Content-Based Analysis of Wikimedia Images2017
- Image-embodied Knowledge Representation Learning2017
- Argument discovery via crowdsourcingThe VLDB Journal, 2017
- Stacked Attention Networks for Image Question Answering2016
- Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding2016
- Effective reinforcement learning following cerebellar damage requires a balance between exploration and motor noiseBrain, 2015
- Return of the Devil in the Details: Delving Deep into Convolutional Nets2014
- A survey on feature selection methodsComputers and Electrical Engineering, 2014
- Freebase2008
- WordNetCommunications of the ACM, 1995