Governing the evidence base: An empirical framework of AI archetypes for policy analysis

Authors:
Open Access
Publisher website
Google Scholar
Abstract
The practice of public administration is confronted by an overwhelming volume of unstructured textual data, creating a critical bottleneck that challenges the capacity for timely, evidence-based policymaking. While large language models (LLMs) offer a powerful solution, the academic and policy discourse on AI governance has primarily focused on the risks associated with quantitative, predictive models, leaving the governance of LLMs, which involve qualitative, interpretive analysis, critically underexplored. This article provides a systematic and empirical answer to this challenge. We evaluate four distinct archetypes for AI-assisted policy analysis: (1) traditional expert-driven analysis, (2) automated topic modelling, (3) human-in-the-loop LLM co-production (ChatGPT with prompting), and (4) institutional-scale API automation (ChatGPT via API), by applying them to a real-world corpus of 63 United Nations policy documents. Our results reveal the inherent trade-offs of each archetype in terms of analytical speed, cost, contextual depth, and the requisite level of human oversight, finding that the human-in-the-loop model offers the most effective balance. We conclude by presenting a durable framework to guide public institutions in making a strategic, evidence-based choice of analytical workflow, ensuring the adoption of AI strengthens, rather than erodes, the integrity of democratic governance.

This publication has 35 references indexed in Scilit: