Pre-Trained Language Model for Temporal Knowledge Graph Completion
Preprint
- 20 November 2024
- preprint
- Published by MDPI AG in Preprints.org
Abstract
Large Language Models (LLMs) have been proven remarkable in natural language processing, which prompts numerous works on knowledge extraction, knowledge fusion, knowledge representation, and knowledge completion. Existing works mainly focus on static multi-relational knowledge graph (KG). Unlike static knowledge graph, temporal knowledge graph (TKG) contains temporal information and evolves over time. Learning and reasoning about the representation of temporal knowledge graph are more difficult. Training or fine-tuning LLMs for temporal graph related tasks incurs significant computational overhead and requires the design of prompts. Conducting tasks of TKG does not necessarily require such complex work. Therefore, we explore temporal knowledge graph completion (TKGC) based on pre-trained "small" language model. We propose TKG-BERT by applying BERT for temporal knowledge graph completion and classification. Specifically, We introduce three ways to model temporal knowledge in TKG-BERT: vanilla knowledge embedding (Van.), explicit time modeling (Exp.) and implicit time modeling (Imp.). TKG-BERT(Van.) only adopts static knowledge without embedding time information; TKG-BERT(Exp.) embeds timestamp in quadruple explicitly; TKG-BERT(Imp.) models time implicitly, by dividing the training set and testing set in chronological order. We conduct experiments on ICEWS14 and ICEWS05-15, which are two public temporal knowledge graph datasets. Various experiments of temporal knowledge graph completion and classification tasks show the effectiveness of pre-trained language model for TKG completion. We also compare the performance of TKG-BERT accross different time modeling way and proportion of training set.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- Leveraging Pre-trained Language Models for Time Interval Prediction in Text-Enhanced Temporal Knowledge Graphs2024
- Relphormer: Relational Graph Transformer for Knowledge Graph RepresentationsNeurocomputing, 2023
- Learning to Walk across Time for Interpretable Temporal Knowledge Graph Completion2021
- Temporal Knowledge Graph Reasoning Based on Evolutional Representation Learning2021
- Timespan-Aware Dynamic Knowledge Graph Embedding by Incorporating Temporal EvolutionIEEE Access, 2020
- DyERNIE: Dynamic Evolution of Riemannian Manifold Embeddings for Temporal Knowledge Graph Completion2020
- Unifying Knowledge Graph Learning and Recommendation: Towards a Better Understanding of User Preferences2019
- A survey of methods for time series change point detectionKnowledge and Information Systems, 2016
- Some Mathematical Notes on Three-Mode Factor AnalysisPsychometrika, 1966
- The Expression of a Tensor or a Polyadic as a Sum of ProductsJournal of Mathematics and Physics, 1927