A Total Error Approach for Validating Event Data
Authors:
- 14 June 2021
- journal article
- research article
- Published by SAGE Publications in American Behavioral Scientist
- Vol. 66 (5) , 603-624
Abstract
Understanding how useful any particular set of event data might be for conflict research requires appropriate methods for assessing validity when ground truth data about the population of interest do not exist. We argue that a total error framework can provide better leverage on these critical questions than previous methods have been able to deliver. We first define a total event data error approach for identifying 19 types of error that can affect the validity of event data. We then address the challenge of applying a total error framework when authoritative ground truth about the actual distribution of relevant events is lacking. We argue that carefully constructed gold standard datasets can effectively benchmark validity problems even in the absence of ground truth data about event populations. To illustrate the limitations of conventional strategies for validating event data, we present a case study of Boko Haram activity in Nigeria over a 3-month offensive in 2015 that compares events generated by six prominent event extraction pipelines—ACLED, SCAD, ICEWS, GDELT, PETRARCH, and the Cline Center’s SPEED project. We conclude that conventional ways of assessing validity in event data using only published datasets offer little insight into potential sources of error or bias. Finally, we illustrate the benefits of validating event data using a total error approach by showing how the gold standard approach used to validate SPEED data offers a clear and robust method for detecting and evaluating the severity of temporal errors in event data.Keywords
This publication has 24 references indexed in Scilit:
- In data we trust? A comparison of UCDP GED and ACLED conflict events datasetsCooperation and Conflict, 2012
- Introducing ACLED: An Armed Conflict Location and Event DatasetJournal of Peace Research, 2010
- Crisis Early Warning and Decision Support: Contemporary Approaches and Thoughts on Future ResearchInternational Studies Review, 2010
- Total Survey Error: Past, Present, and FuturePublic Opinion Quarterly, 2010
- Total Survey Error: Design, Implementation, and EvaluationPublic Opinion Quarterly, 2010
- The Use of Newspaper Data in the Study of Collective ActionAnnual Review of Sociology, 2004
- The Concept of Validity.Psychological Review, 2004
- Using Media-Based Data in Studies of PoliticsAmerican Journal of Political Science, 2000
- Validity Assessment of a Machine-Coded Event Data Set for the Middle East, 1982-92American Journal of Political Science, 1994
- Construct validity in psychological tests.Psychological Bulletin, 1955