The Choice of Textual Knowledge Base in Automated Claim Checking
Open Access
- 2 March 2023
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in Journal of Data and Information Quality
- Vol. 15 (1) , 1-22
Abstract
Automated claim checking is the task of determining the veracity of a claim given evidence retrieved from a textual knowledge base of trustworthy facts. While previous work has taken the knowledge base as given and optimized the claim-checking pipeline, we take the opposite approach—taking the pipeline as given, we explore the choice of the knowledge base. Our first insight is that a claim-checking pipeline can be transferred to a new domain of claims with access to a knowledge base from the new domain. Second, we do not find a “universally best” knowledge base—higher domain overlap of a task dataset and a knowledge base tends to produce better label accuracy. Third, combining multiple knowledge bases does not tend to improve performance beyond using the closest-domain knowledge base. Finally, we show that the claim-checking pipeline’s confidence score for selecting evidence can be used to assess whether a knowledge base will perform well for a new set of claims, even in the absence of ground-truth labels.Keywords
All Related Versions
This publication has 29 references indexed in Scilit:
- The spread of true and false news onlineScience, 2018
- Fully Automated Fact Checking Using External Sources2017
- The knowledge graph as the default data model for learning on heterogeneous knowledgeData Science, 2017
- Anserini2017
- Credibility Assessment of Textual Claims on the Web2016
- Emergent: a novel data-set for stance classification2016
- Fact Checking: Task definition and dataset construction2014
- In Kidney Transplant Patients, Alemtuzumab but Not Basiliximab/Low-Dose Rabbit Anti-Thymocyte Globulin Induces B Cell Depletion and Regeneration, Which Associates with a High Incidence of De Novo Donor-Specific Anti-HLA Antibody DevelopmentThe Journal of Immunology, 2013
- Fact checking and analyzing the web2013
- Induction Therapy With Autologous Mesenchymal Stem Cells in Living-Related Kidney TransplantsJAMA, 2012