Incorporating Structured Sentences with Time-enhanced BERT for Fully-inductive Temporal Relation Prediction
Authors:
- Zhongwu ChenZCZhongwu Chen
- Chengjin XuCXChengjin Xu
- Fenglong SuFSFenglong Su
- Zhen HuangZHZhen Huang
- Yong DouYDYong Dou
Preprint
Abstract
Temporal relation prediction in incomplete temporal knowledge graphs (TKGs) is a popular temporal knowledge graph completion (TKGC) problem in both transductive and inductive settings. Traditional embedding-based TKGC models (TKGE) rely on structured connections and can only handle a fixed set of entities, i.e., the transductive setting. In the inductive setting where test TKGs contain emerging entities, the latest methods are based on symbolic rules or pre-trained language models (PLMs). However, they suffer from being inflexible and not time-specific, respectively. In this work, we extend the fully-inductive setting, where entities in the training and test sets are totally disjoint, into TKGs and take a further step towards a more flexible and time-sensitive temporal relation prediction approach SST-BERT, incorporating Structured Sentences with Time-enhanced BERT. Our model can obtain the entity history and implicitly learn rules in the semantic space by encoding structured sentences, solving the problem of inflexibility. We propose to use a time masking MLM task to pre-train BERT in a corpus rich in temporal tokens specially generated for TKGs, enhancing the time sensitivity of SST-BERT. To compute the probability of occurrence of a target quadruple, we aggregate all its structured sentences from both temporal and semantic perspectives into a score. Experiments on the transductive datasets and newly generated fully-inductive benchmarks show that SST-BERT successfully improves over state-of-the-art baselines.Keywords
All Related Versions
This publication has 25 references indexed in Scilit:
- Meta-Learning Based Knowledge Extrapolation for Temporal Knowledge Graph2023
- Time-aware Entity Alignment using Temporal Relational Attention2022
- Open-world knowledge graph completion for unseen entities and relations via attentive feature aggregationInformation Sciences, 2022
- Geometric Algebra based Embeddings for Static and Temporal Knowledge Graph CompletionIEEE Transactions on Knowledge and Data Engineering, 2022
- Time Masking for Temporal Language Models2022
- A Survey on Knowledge Graphs: Representation, Acquisition, and ApplicationsIEEE Transactions on Neural Networks and Learning Systems, 2021
- CokeBERT: Contextual knowledge selection and embedding towards enhanced pre-trained language modelsAI Open, 2021
- Deriving Validity Time in Knowledge Graph2018
- Introducing Wikidata to the Linked Data Web2014
- Yago2007