Prompt selection matters: enhancing text annotations for social sciences with large language models

Authors:
Open Access
Publisher website
Google Scholar
Abstract
Large Language Models have recently been applied to text annotation tasks from social sciences, equating or surpassing the performance of human workers at a fraction of the cost. However, very few inquiries in the social sciences have been made of the impact of prompt selection on labelling accuracy. In this study, we show that performance greatly varies between prompts, and we apply the method of automatic prompt optimization to systematically craft high quality prompts. We also provide the community with a simple, browser-based implementation of the method at https://prompt-ultra.github.io/.
Funding Information
  • Université Paris-Saclay

This publication has 34 references indexed in Scilit: