SYNCode: Synergistic Human–LLM Collaboration for Enhanced Data Annotation in Stack Overflow
Open Access
- 9 May 2025
- journal article
- research article
- Published by MDPI AG in Information
- Vol. 16 (5) , 392
Abstract
Large language models (LLMs) have rapidly advanced natural language processing, showcasing remarkable effectiveness as automated annotators across various applications. Despite their potential to significantly reduce annotation costs and expedite workflows, annotations produced solely by LLMs can suffer from inaccuracies and inherent biases, highlighting the necessity of maintaining human oversight. In this article, we present a synergistic human–LLM collaboration approach for data annotation enhancement (SYNCode). This framework is designed explicitly to facilitate collaboration between humans and LLMs for annotating complex, code-centric datasets such as Stack Overflow. The proposed approach involves an integrated pipeline that initially employs TF-IDF analysis for quick identification of relevant textual elements. Subsequently, we leverage advanced transformer-based models, specifically NLP Transformer and UniXcoder, to capture nuanced semantic contexts and code structures, generating more accurate preliminary annotations. Human annotators then engage in iterative refinement, validating and adjusting annotations to enhance accuracy and mitigate biases introduced during automated labeling. To operationalize this synergistic workflow, we developed the SYNCode prototype, featuring an interactive graphical interface that supports real-time collaborative annotation between humans and LLMs. This enables annotators to iteratively refine and validate automated suggestions effectively. Our integrated human–LLM collaborative methodology demonstrates considerable promise in achieving high-quality, reliable annotations, particularly for domain-specific and technically demanding datasets, thereby enhancing downstream tasks in software engineering and natural language processing.Keywords
Funding Information
- Nebraska Collaboration Initiative (NRI-47130)
This publication has 34 references indexed in Scilit:
- Can Large Language Models Transform Computational Social Science?Computational Linguistics, 2024
- Evaluating GPT-3 Generated Explanations for Hateful Content ModerationarXiv, 2023
- ChatGPT outperforms crowd workers for text-annotation tasksProceedings of the National Academy of Sciences, 2023
- SeisLabel: An AI-Assisted Annotation Tool for Seismic Data Labeling2023
- Large Language Models for Software Engineering: Survey and Open Problems2023
- Fundamentals of Artificial Intelligence2020
- Improving Human-Machine Cooperative Visual Search With Soft HighlightingACM Transactions on Applied Perception, 2017
- PAL, a tool for Pre-annotation and Active LearningJournal for Language Technology and Computational Linguistics, 2016
- Evaluating the effects of machine pre-annotation and an interactive annotation interface on manual de-identification of clinical textJournal of Biomedical Informatics, 2014
- Evaluating the impact of pre-annotation on annotation speed and potential bias: natural language processing gold standard development for clinical named entity recognition in clinical trial announcementsJournal of the American Medical Informatics Association, 2014