Assessing the Impact of Conspiracy Theories Using Large Language Models
Preprint
Abstract
Measuring the relative impact of CTs is important for prioritizing responses and allocating resources effectively, especially during crises. However, assessing the actual impact of CTs on the public poses unique challenges. It requires not only the collection of CT-specific knowledge but also diverse information from social, psychological, and cultural dimensions. Recent advancements in large language models (LLMs) suggest their potential utility in this context, not only due to their extensive knowledge from large training corpora but also because they can be harnessed for complex reasoning. In this work, we develop datasets of popular CTs with human-annotated impacts. Borrowing insights from human impact assessment processes, we then design tailored strategies to leverage LLMs for performing human-like CT impact assessments. Through rigorous experiments, we textit{discover that an impact assessment mode using multi-step reasoning to analyze more CT-related evidence critically produces accurate results; and most LLMs demonstrate strong bias, such as assigning higher impacts to CTs presented earlier in the prompt, while generating less accurate impact assessments for emotionally charged and verbose CTs.Keywords
All Related Versions
This publication has 20 references indexed in Scilit:
- GPT-4V(ision) as A Social Media Analysis EngineACM Transactions on Intelligent Systems and Technology, 2025
- Can Large Language Models Provide Useful Feedback on Research Papers? A Large-Scale Empirical AnalysisNEJM AI, 2024
- Online conspiracy communities are more resilient to deplatformingPNAS Nexus, 2023
- An epidemic of uncertainty: rumors, conspiracy theories and vaccine hesitancyNature Medicine, 2022
- De-platforming disinformation: conspiracy theories and their controlInformation, Communication & Society, 2021
- Conspiracy in the time of corona: automatic detection of emerging COVID-19 conspiracy theories in social media and the newsJournal of Computational Social Science, 2020
- Conspiracy theories as barriers to controlling the spread of COVID-19 in the U.S.Social Science & Medicine, 2020
- Pylons ablaze: Examining the role of 5G COVID‐19 conspiracy beliefs and support for violenceBritish Journal of Social Psychology, 2020
- Coronavirus conspiracy beliefs, mistrust, and compliance with government guidelines in EnglandPsychological Medicine, 2020
- Using social and behavioural science to support COVID-19 pandemic responseNature Human Behaviour, 2020