Scaling Up With Integrity: Valid and Efficient Narrative Policy Framework Analyses Using Large Language Models
- 9 June 2025
- journal article
- Published by Wiley in Policy Studies Journal
- Vol. 54 (2)
Abstract
Given vast quantities of digital and online data—such as news articles, congressional testimony, and social media posts—the potential for large scale narrative analyses has dramatically increased. Narrative Policy Framework (NPF) researchers can now access thousands or even millions of policy‐relevant documents. However, these large datasets also bring new challenges, including how to scale analyses while maintaining validity and reliability. Traditionally, NPF studies employ labor intensive human coding to identify narrative elements, limiting the number of documents that can be feasibly analyzed. Recognizing the potential of large language models (LLMs) to automate such analyses, this study demonstrates the use of zero‐shot, few‐shot, and fine‐tuned classification techniques to identify narrative elements. By prioritizing construct validity at each step in the process, we offer a rigorous and replicable approach to integrating LLMs into narrative policy research.Keywords
This publication has 27 references indexed in Scilit:
- A survey on large language model (LLM) security and privacy: The Good, The Bad, and The UglyHigh-Confidence Computing, 2024
- Practical and ethical challenges of large language models in education: A systematic scoping reviewBritish Journal of Educational Technology, 2023
- Relatio: Text Semantics Capture Political and Economic NarrativesPolitical Analysis, 2023
- ChatGPT: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scopeInternet of Things and Cyber-Physical Systems, 2023
- Strategically constructed narratives on artificial intelligence: What stories are told in governmental artificial intelligence policies?Government Information Quarterly, 2022
- Trump's tweets as policy narratives: Constructing the immigration issue via social mediaPolitics & Policy, 2022
- Talking Policy in Congressional Campaigns: Construction of Policy Narratives in Electoral PoliticsPolitics & Policy, 2020
- Building thesaurus lexicon using dictionary-based approach for sentiment classification2016
- Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political TextsPolitical Analysis, 2013
- Applied Logistic Regression2000