How well do collaboration quality estimation models generalize across authentic school contexts?
- 20 October 2023
- journal article
- research article
- Published by Wiley in British Journal of Educational Technology
- Vol. 55 (4) , 1602-1624
Abstract
Multimodal learning analytics (MMLA) research has made significant progress in modelling collaboration quality for the purpose of understanding collaboration behaviour and building automated collaboration estimation models. Deploying these automated models in authentic classroom scenarios, however, remains a challenge. This paper presents findings from an evaluation of collaboration quality estimation models. We collected audio, video and log data from two different Estonian schools. These data were used in different combinations to build collaboration estimation models and then assessed across different subjects, different types of activities (collaborative‐writing, group‐discussion) and different schools. Our results suggest that the automated collaboration model can generalize to the context of different schools but with a 25% degradation in balanced accuracy (from 82% to 57%). Moreover, the results also indicate that multimodality brings more performance improvement in the case of group‐discussion‐based activities than collaborative‐writing‐based activities. Further, our results suggest that the video data could be an alternative for understanding collaboration in authentic settings where higher‐quality audio data cannot be collected due to contextual factors. The findings have implications for building automated collaboration estimation systems to assist teachers with monitoring their collaborative classrooms. Practitioners notes: What is already known about this topic: Multimodal learning analytics researchers have established several features as potential indicators for collaboration quality, e.g., speaking time or joint visual attention. The current state of the art has shown the feasibility of building automated collaboration quality models. Recent research has provided preliminary evidence of the generalizability of developed automated models across contexts different in terms of given task and subject. What does this paper add: This paper offers collaboration indicators for different types of collaborative learning activities in authentic classroom settings. The paper includes a systematic investigation into collaboration quality automated model's generalizability across different tasks, types of tasks and schools. This paper also offers a comparison between different modalities' potential to estimate collaboration quality in authentic settings. Implications for practice: The findings inform the development of automated collaboration monitoring systems for authentic classroom settings. This paper provides evidence on across‐school generalizability capabilities of collaboration quality estimation models.Keywords
This publication has 42 references indexed in Scilit:
- Modelling Symmetry of Activity as an Indicator of Collocated Group Collaboration2011
- Analyzing Collaborative Interactions Across Domains and Settings: An Adaptable Rating Scheme2010
- The teacher's role in promoting collaborative dialogue in the classroomBritish Journal of Educational Psychology, 2009
- Emote aloud during learning with AutoTutor: Applying the Facial Action Coding System to cognitive–affective states during learningCognition and Emotion, 2008
- A framework to analyze argumentative knowledge construction in computer-supported collaborative learningComputers & Education, 2006
- Learning to Collaborate: An Instructional Approach to Promoting Collaborative Problem Solving in Computer-Mediated SettingsJournal of the Learning Sciences, 2005
- When Smart Groups FailJournal of the Learning Sciences, 2003
- How collaborative is pair work? ESL tertiary students composing in pairsLanguage Teaching Research, 2001
- When teams do not function the way they ought toInternational Journal of Educational Research, 1989
- Some signals and rules for taking speaking turns in conversations.Journal of Personality and Social Psychology, 1972