Tracing the Impact of Bias in Link Prediction
Open Access
- 8 April 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1626-1633
Abstract
Link prediction (LP) in knowledge graphs (KGs) uses embedding-based approaches and machine learning (ML) models to uncover new facts. In order to not overstate and accurately assess the performance of these techniques, their comprehensive and rigorous evaluation is needed. In this work, we suggest a framework to systematically trace and analyze bias---specifically, test leakage bias and sample selection bias---in training and testing knowledge graphs (KGs). The goal is to evaluate how bias affects the performance of LP models We specify a collection of bias measures in SPARQL (the W3C standard query language) to facilitate the analysis of any RDF graph with regard to its structural bias properties. Further, we evaluate our framework over seven state-of-the-art LP datasets (e.g., FB15k-237, WN18RR, and YAGO3-10) and the TransE model. Our findings show how bias, i.e., overrepresentation of entities and relations and pronounced information redundancy, is present across all datasets and how it advantageously impacts the reported performance of the LP model. With these results, we make a call for thorough assessments of data sources in order to discourage the use of biased datasets when appropriate, and to also help improve our understanding of how LP models work and to better interpret their produced output.Keywords
Funding Information
- HORIZON EUROPE Marie Sklodowska-Curie Actions (860630)
This publication has 30 references indexed in Scilit:
- Knowledge graphs as tools for explainable machine learning: A surveyArtificial Intelligence, 2022
- Do Embeddings Actually Capture Knowledge Graph Semantics?2021
- Realistic Re-evaluation of Knowledge Graph Completion Methods: An Experimental Study2020
- Model Cards for Model Reporting2019
- WikidataCommunications of the ACM, 2014
- Link Prediction2011
- Freebase2008
- Yago2007
- DBpedia: A Nucleus for a Web of Open Data2007
- WordNetCommunications of the ACM, 1995