Enhancing Text Annotation Through Rationale-Driven Collaborative Few-Shot Prompting
Authors:
- 6 April 2025
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-5
Abstract
The traditional data annotation process is often labor-intensive, time-consuming, and susceptible to human bias, which complicates the management of increasingly complex datasets. This study explores the potential of large language models (LLMs) as automated data annotators to improve efficiency and consistency in annotation tasks. By employing rationale-driven collaborative few-shot prompting techniques, we aim to improve the performance of LLMs in text annotation. We conduct a rigorous evaluation of five LLMs across four benchmark datasets, comparing seven distinct methodologies. Our results demonstrate that collaborative methods consistently outperform traditional few-shot techniques and other baseline approaches, particularly in complex annotation tasks. Our work provides valuable insights and a robust framework for leveraging collaborative learning methods to tackle challenging text annotation tasks.Keywords
All Related Versions
This publication has 13 references indexed in Scilit:
- 🧜Siren’s Song in the AI Ocean: A Survey on Hallucination in Large Language ModelsComputational Linguistics, 2025
- A Comparative Study on Annotation Quality of Crowdsourcing and LLm Via Label Aggregation2024
- Character Attribute Extraction from Movie Scripts Using LLMs2024
- PDFChatAnnotator: A Human-LLM Collaborative Multi-Modal Data Annotation Tool for PDF-Format Catalogs2024
- Large Language Models for Data Annotation and Synthesis: A Survey2024
- ChatGPT outperforms crowd workers for text-annotation tasksProceedings of the National Academy of Sciences of the United States of America, 2023
- Evaluation of ChatGPT on Biomedical Tasks: A Zero-Shot Comparison with Fine-Tuned Generative Transformers2023
- Can Large Language Models Be an Alternative to Human Evaluations?2023
- Is GPT-3 a Good Data Annotator?2023
- Improving In-Context Few-Shot Learning via Self-Supervised Training2022