Disagreement as Data: Reasoning Trace Analytics in Multi-Agent Systems
Top Cited Papers
Open Access
- 26 April 2026
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 933-939
Abstract
Learning analytics researchers often analyze qualitative student data such as coded annotations or interview transcripts to understand learning processes. With the rise of generative AI, fully automated and human–AI workflows have emerged as promising methods for analysis. However, methodological standards to guide such workflows remain limited. In this study, we propose that reasoning traces generated by large language model (LLM) agents, especially within multi-agent systems, constitute a novel and rich form of process data to enhance interpretive practices in qualitative coding. We apply cosine similarity to LLM reasoning traces to systematically detect, quantify, and interpret disagreements among agents, reframing disagreement as a meaningful analytic signal. Analyzing nearly 10,000 instances of agent pairs coding human tutoring dialog segments, we show that LLM agents’ semantic reasoning similarity robustly differentiates consensus from disagreement and correlates with human coding reliability. Qualitative analysis guided by this metric reveals nuanced instructional sub-functions within codes and opportunities for conceptual codebook refinement. By integrating quantitative similarity metrics with qualitative review, our method bears potential to improve and accelerate the process of establishing inter-rater reliability during coding by surfacing interpretive ambiguity, especially when LLMs collaborate with humans. We discuss how reasoning-trace disagreements represent a valuable new class of analytic signals advancing methodological rigor and interpretive depth in educational research.Keywords
All Related Versions
This publication has 25 references indexed in Scilit:
- Artificial intelligence in multimodal learning analytics: A systematic literature reviewComputers and Education: Artificial Intelligence, 2025
- (How) Do reasoning models reason?Annals of the New York Academy of Sciences, 2025
- ChatGPT for Education Research: Exploring the Potential of Large Language Models for Qualitative Codebook Development2024
- Students’ reflection on the effect of collaborative learning on the learning environment and academic achievement in online reflective platformsReflective Practice, 2024
- Detector-driven classroom interviewing: focusing qualitative researcher time by selecting cases in situEducational Technology Research and Development, 2023
- From nCoder to ChatGPT: From Automated Coding to Refining Human Coding2023
- Leveraging natural language processing to support automated assessment and feedback for student open responses in mathematicsJournal of Computer Assisted Learning, 2023
- One size fits all? What counts as quality practice in (reflexive) thematic analysis?Qualitative Research in Psychology, 2020
- Cosine similarity to determine similarity measure: Study case in online essay assessment2016
- Multimodal learning analytics2013