Towards Semantic Consistency: Dirichlet Energy Driven Robust Multi-Modal Entity Alignment
Authors:
- 13 May 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 3559-3572
Abstract
Multi-Modal Entity Alignment (MMEA) is a pivotal task in Multi-Modal Knowledge Graphs (MMKGs), seeking to identify identical entities by leveraging associated modal attributes. However, real-world MMKGs confront the challenges of semantic inconsistency arising from diverse and incomplete data sources. This inconsistency is predominantly caused by the absence of specific modal attributes, manifesting in two distinct forms: disparities in attribute counts or the absence of certain modalities. Current methods address these issues through attribute interpolation, but their reliance on predefined distributions introduces modality noise, compromising original semantic information. Furthermore, the absence of a generalizable theoretical principle hampers progress towards achieving semantic consistency. In this work, we propose a generalizable theoretical principle by examining semantic consistency from the perspective of Dirichlet energy. Our research reveals that, in the presence of semantic inconsistency, models tend to overfit to modality noise, leading to over-smoothing and performance oscillations or declines, particularly in scenarios with a high rate of missing modality. To overcome these challenges, we propose DESAlign, a robust method addressing the over-smoothing caused by semantic inconsistency and interpolating missing semantics using existing modalities. Specifically, we devise a training strategy for multi-modal knowledge graph learning based on our proposed principle. Then, we introduce a propagation strategy that utilizes existing features to provide interpolation solutions for missing semantic features. DESAlign outperforms existing approaches across 60 benchmark splits, encompassing both monolingual and bilingual scenarios, achieving state-of-the-art performance. Experiments on splits with high missing modal attributes demonstrate its effectiveness, providing a robust MMEA solution to semantic inconsistency in real-world MMKGs.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (62201072,U23B2001,62171057,62101064,62001054,62071067,MCM20200202,MCM20180101)
This publication has 49 references indexed in Scilit:
- Rethinking Uncertainly Missing and Ambiguous Visual Modality in Multi-Modal Entity Alignment2023
- MultiJAF: Multi-modal joint entity alignment framework for multi-modal knowledge graphNeurocomputing, 2022
- Multi-modal entity alignment in hyperbolic spaceNeurocomputing, 2021
- MMEA: Entity Alignment for Multi-modal Knowledge Graph2020
- Emotion recognition using multi-modal data and machine learning techniques: A tutorial and reviewInformation Fusion, 2020
- MMKG: Multi-modal Knowledge Graphs2019
- Unsupervised Entity Alignment Using Attribute Triples and Relation Triples2019
- Cross-Lingual Entity Alignment via Joint Attribute-Preserving Embedding2017
- LogMap: Logic-Based and Scalable Ontology Matching2011
- Spectral Graph Theory1996