Fine-Tuned Large Language Models for Theory-Aligned Social Science Content Analysis: Evidence From Collaborative Learning Discourse
- 12 November 2026
- journal article
- Published by SAGE Publications in Social Science Computer Review
Abstract
Theory-based content analysis is central to social science research, but coding context-dependent interactional discourse remains difficult to scale. This study examines whether fine-tuned large language models (LLMs) can support theory-aligned, episode-level coding of complex social interaction, using socially shared metacognitive regulation (SSMR) in collaborative learning discourse as a demanding empirical case. Using a human-coded dataset of 2,372 discourse episodes from undergraduate collaborative problem-solving, we adapted the large language model content analysis (LACA) framework through an AI-aligned codebook, prompt design, and supervised fine-tuning. Prompt-only and fine-tuned models were compared across episode-level classification and multilayer SSMR coding. Fine-tuned models substantially improved agreement with human coding and predictive performance relative to prompt-only baselines. Performance was strongest for episode-level classification and several higher-support coding layers, whereas errors remained concentrated in sparse, boundary-prone, and highly interpretive categories requiring social-intent reconstruction and temporally unfolding context. The findings extend recent work on LLM-assisted qualitative and content analysis by showing both the promise and limits of supervised, theory-aligned automation for multilayer social interaction coding. Overall, the study supports a human-centered approach to LLM-assisted content analysis, in which automated coding can support more stable coding tasks while human expertise remains essential for ambiguous, boundary-prone, and interpretively demanding judgments.Funding Information
- Hanyang University (HY-202600000001877)
This publication has 35 references indexed in Scilit:
- Assessing the Potential and Limits of Large Language Models in Qualitative Coding2024
- Evaluating the performance of ChatGPT and GPT-4o in coding classroom discourse data: A study of synchronous online mathematics instructionComputers and Education: Artificial Intelligence, 2024
- ChatGPT for Education Research: Exploring the Potential of Large Language Models for Qualitative Codebook Development2024
- Augmenting assessment with AI coding of online student discourse: A question of reliabilityComputers and Education: Artificial Intelligence, 2024
- Automated coding of student chats, a trans-topic and language approachComputers and Education: Artificial Intelligence, 2023
- A systematic review of artificial intelligence techniques for collaborative learning over the past two decadesComputers and Education: Artificial Intelligence, 2022
- Significance of forms and foci of metacognitive regulation in collaborative science learning of less and more successful outcome groups in diverse contextsInstructional Science, 2021
- Variations in socially shared metacognitive regulation and their relation with university students’ performanceMetacognition and Learning, 2020
- Tutor learning: the role of explaining and responding to questionsInstructional Science, 2007
- A framework to analyze argumentative knowledge construction in computer-supported collaborative learningComputers & Education, 2006