GPT models for text annotation: An empirical exploration in public policy research
Authors:
- 2 May 2025
- journal article
- Published by Wiley in Policy Studies Journal
- Vol. 54 (1)
Abstract
Text annotation, the practice of labeling text following a predetermined scheme, is essential to qualitative public policy research. Despite its importance, annotating large qualitative data faces challenges of high labor and time costs. Recent developments in large language models (LLMs), specifically models with generative pretrained transformers (GPTs), show a potential approach that may alleviate the burden of manual text annotation. In this report, we first introduce a small sample pretest strategy for researchers to decide whether to use Open AI's GPT models for text annotation. In addition, we test if GPT models can substitute human coders by comparing the results of two GPT models with different prompting strategies against human annotation. Using email messages collected from a national corresponding experiment in the US nursing home market as an example, on average, we demonstrate 86.25% percentage agreement between GPT and human annotations. We also show that GPT models possess context‐based limitations. Our report ends with reflections and suggestions for readers who are interested in using GPT models for text annotation.Keywords
All Related Versions
This publication has 28 references indexed in Scilit:
- ChatGPT as Research Scientist: Probing GPT’s capabilities as a Research Librarian, Research Ethicist, Data Generator, and Data PredictorProceedings of the National Academy of Sciences, 2024
- GPT is an effective tool for multilingual psychological text analysisProceedings of the National Academy of Sciences, 2024
- Let the algorithm speak: How to use neural networks for automatic item generation in psychological scale development.Psychological Methods, 2024
- The Effectiveness of GPT-4 as Financial News Annotator Versus Human Annotator in Improving the Accuracy and Performance of Sentiment Analysis2024
- The Language of Creativity: Evidence from Humans and Large Language ModelsThe Journal of Creative Behavior, 2024
- Systematic testing of three Language Models reveals low language accuracy, absence of response stability, and a yes-response biasProceedings of the National Academy of Sciences, 2023
- Will Affective Computing Emerge From Foundation Models and General Artificial Intelligence? A First Evaluation of ChatGPTIEEE Intelligent Systems, 2023
- Using cognitive psychology to understand GPT-3Proceedings of the National Academy of Sciences, 2023
- Machine Learning for Public Administration Research, With Application to Organizational ReputationJournal of Public Administration Research and Theory, 2018
- Thematic analysis.2012