Language Models-enhanced Semantic Topology Representation Learning For Temporal Knowledge Graph Extrapolation
Open Access
- 21 October 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 3227-3236
Abstract
Temporal Knowledge Graph (TKG) extrapolation aims to predict future missing facts based on historical information, which has exhibited both semantics and topology of events. The mainstream methods have advanced the prediction performance by exploring the potential of topology representations of TKGs based on dedicated temporal Graph Neural Networks (GNNs). Until recently, few Language Models (LM) based methods have attempted to model the semantic representations of TKGs, however, lacking specific designs for the topology information. Therefore, we propose a Semantic TOpology REpresentation learning (STORE) framework enhanced by LMs to bridge the gap between the semantics and topology of TKGs. Firstly, we tackle the challenge of long historical facts modeling by a time-aware sampling based on semantic priors to extract concise yet precise facts. Secondly, we handle the challenge of the interaction between topology and semantics by transforming graph representations into virtual tokens that are then integrated with generated prompts and fed into LMs. Finally, multi-head attention is adopted to obtain better semantic topology representations, thereby achieving joint optimization of both temporal GNNs and LMs. Extensive experiments on five datasets show that our STORE outperforms state-of-the-art GNNs- and LM-based methods.Keywords
Funding Information
- Ningbo Natural Science Foundation (2023J281)
- Zhejiang Province High-Level Talents Special Support Program Leading Talent of Technological Innovation of Ten-Thousands Talents Program (2022R52046)
- Zhejiang Province JianBingLingYan+X Research and Development Plan (2024C01114)
- the Joint Funds of the Zhejiang Provincial Natural Science Foundation of China (LHZSD24F020001)
This publication has 17 references indexed in Scilit:
- Learning Long- and Short-term Representations for Temporal Knowledge Graph Reasoning2023
- Temporal Aggregation and Propagation Graph Neural Networks for Dynamic RepresentationIEEE Transactions on Knowledge and Data Engineering, 2023
- RETIA: Relation-Entity Twin-Interact Aggregation for Temporal Knowledge Graph Extrapolation2023
- Transition Propagation Graph Neural Networks for Temporal NetworksIEEE Transactions on Neural Networks and Learning Systems, 2022
- Financial time series forecasting with multi-modality graph neural networkPattern Recognition, 2022
- Learning Dynamic Preference Structure Embedding From Temporal Networks2021
- Temporal Knowledge Graph Reasoning Based on Evolutional Representation Learning2021
- Dynamic Heterogeneous Graph Neural Network for Real-time Event Prediction2020
- Structured Sequence Modeling with Graph Convolutional Recurrent Networks2018
- Deriving Validity Time in Knowledge Graph2018