Mining an "anti-knowledge base" from Wikipedia updates with applications to fact checking and beyond
Authors:
Open Access
- 9 December 2019
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in Proceedings of the VLDB Endowment
- Vol. 13 (4) , 561-573
Abstract
We introduce the problem of anti-knowledge mining. Our goal is to create an "anti-knowledge base" that contains factual mistakes. The resulting data can be used for analysis, training, and benchmarking in the research domain of automated fact checking. Prior data sets feature manually generated fact checks of famous misclaims. Instead, we focus on the long tail of factual mistakes made by Web authors, ranging from erroneous sports results to incorrect capitals.We mine mistakes automatically, by an unsupervised approach, from Wikipedia updates that correct factual mistakes. Identifying such updates (only a small fraction of the total number of updates) is one of the primary challenges. We mine anti-knowledge by a multi-step pipeline. First, we filter out candidate updates via several simple heuristics. Next, we correlate Wikipedia updates with other statements made on the Web. Using claim occurrence frequencies as input to a probabilistic model, we infer the likelihood of corrections via an iterative expectation-maximization approach. Finally, we extract mistakes in the form of subject-predicate-object triples and rank them according to several criteria. Our end result is a data set containing over 110,000 ranked mistakes with a precision of 85% in the top 1% and a precision of over 60% in the top 25%. We demonstrate that baselines achieve significantly lower precision. Also, we exploit our data to verify several hypothesis on why users make mistakes. We finally show that the AKB can be used to find mistakes on the entire Web.Keywords
This publication has 14 references indexed in Scilit:
- Toward Automated Fact-Checking2017
- Identifying Semantic Edit Intentions from Revisions in Wikipedia2017
- On the Retrieval of Wikipedia Articles Containing Claims on Controversial Topics2016
- Mining Subjective Properties on the Web2015
- Leveraging Linguistic Structure For Open Domain Information Extraction2015
- Resolving conflicts in heterogeneous data by truth discovery and source reliability estimation2014
- Truth Discovery with Multiple Conflicting Information Providers on the WebIEEE Transactions on Knowledge and Data Engineering, 2008
- Truth discovery with multiple conflicting information providers on the web2007
- Web-scale information extraction in knowitall2004
- Maximum Likelihood Estimation of Observer Error-Rates Using the EM AlgorithmJournal of the Royal Statistical Society Series C: Applied Statistics, 1979