The COVID-19 Infodemic
Authors:
Open Access
- 19 October 2020
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1305-1314
Abstract
Misinformation is an ever increasing problem that is difficult to solve for the research community and has a negative impact on the society at large. Very recently, the problem has been addressed with a crowdsourcing-based approach to scale up labeling efforts: to assess the truthfulness of a statement, instead of relying on a few experts, a crowd of (non-expert) judges is exploited. We follow the same approach to study whether crowdsourcing is an effective and reliable method to assess statements truthfulness during a pandemic. We specifically target statements related to the COVID-19 health emergency, that is still ongoing at the time of the study and has arguably caused an increase of the amount of misinformation that is spreading online (a phenomenon for which the term "infodemic" has been used). By doing so, we are able to address (mis)information that is both related to a sensitive and personal issue like health and very recent as compared to when the judgment is done: two issues that have not been analyzed in related work. In our experiment, crowd workers are asked to assess the truthfulness of statements, as well as to provide evidence for the assessments as a URL and a text justification. Besides showing that the crowd is able to accurately judge the truthfulness of the statements, we also report results on many different aspects, including: agreement among workers, the effect of different aggregation functions, of scales transformations, and of workers background / bias. We also analyze workers behavior, in terms of queries submitted, URLs found / selected, text justifications, and other behavioral data like clicks and mouse actions collected by means of an ad hoc logger.Keywords
All Related Versions
Funding Information
- Facebook Research (The Online Safety Benchmark)
- Australian Research Council (DP190102141)
- Region Friuli ? Venezia Giulia (1619942002 / 1420AFPLO1)
This publication has 20 references indexed in Scilit:
- Homogeneity-Based Transmissive Process to Model True and False News in Social Networks2019
- All Those Wasted Hours2019
- Overview of the CLEF-2018 CheckThat! Lab on Automatic Identification and Verification of Political Claims2018
- On Fine-Grained Relevance Scales2018
- Considering Assessor Agreement in IR Evaluation2017
- "Liar, Liar Pants on Fire": A New Benchmark Dataset for Fake News Detection2017
- Why Students Share Misinformation on Social Media: Motivation, Gender, and Study-level DifferencesThe Journal of Academic Librarianship, 2015
- Statistical quality estimation for general crowdsourcing tasks2013
- Cognitive Reflection and Decision MakingJournal of Economic Perspectives, 2005
- MAGNITUDE ESTIMATION: NOTES ON WHAT, HOW, WHEN, AND WHY TO USE ITJournal of Food Quality, 1977