Detecting pro-kremlin disinformation using large language models
Open Access
- 1 April 2025
- journal article
- Published by SAGE Publications in Research & Politics
- Vol. 12 (2)
Abstract
A growing body of literature examines manipulative information by detecting political mis-/disinformation in text data. This line of research typically involves highly costly manual annotation of text for manual content analysis, and/or training and validating automated downstream approaches. We examine whether Large Language Models (LLMs) can detect pro-Kremlin disinformation about the war in Ukraine, focusing on the case of the downing of the civilian flight MH17. We benchmark methods using a large set of tweets labeled by expert annotators. We show that both open and closed LLMs can accurately detect pro-Kremlin disinformation tweets, outperforming both a research assistant and supervised models used in earlier research and at drastically lower cost compared to either research assistants or crowd workers. Our findings contribute to the literature on mis/-disinformation by showcasing how LLMs can substantially lower the costs of detection even when the labeling requires complex, context-specific knowledge about a given case.Keywords
This publication has 15 references indexed in Scilit:
- Using proprietary language models in academic research requires explicit justificationNature Computational Science, 2023
- ChatGPT outperforms crowd workers for text-annotation tasksProceedings of the National Academy of Sciences of the United States of America, 2023
- Less Annotating, More Classifying: Addressing the Data Scarcity Issue of Supervised Machine Learning with Deep Transfer Learning and BERT-NLIPolitical Analysis, 2023
- The shape of and solutions to the MTurk quality crisisPolitical Science Research and Methods, 2020
- Ideological Asymmetry in the Reach of Pro-Russian Digital Disinformation to United States AudiencesJournal of Communication, 2019
- Fake news on Twitter during the 2016 U.S. presidential electionScience, 2019
- State, media and civil society in the information warfare over Ukraine: citizen curators of digital disinformationInternational Affairs, 2018
- Detecting Bots on Russian Political TwitterBig Data, 2017
- Political Fact-Checking on Twitter: When Do Corrections Have an Effect?Political Communication, 2017
- Crowd-sourced Text Analysis: Reproducible and Agile Production of Political DataAmerican Political Science Review, 2016