Hanall Sung

In various technology-enhanced learning (TEL) environments, knowledge co-creation progresses through multimodal interactions that integrate verbal and nonverbal modalities, such as speech and gestures. This study investigated two distinct analytical approaches for analyzing multimodal interactions—triangulating and interleaving—by applying them to collaborative learning processes during an online embodied mathematics intervention. The findings demonstrate that the interleaving approach captures the temporal dynamics and nuanced interplay between multimodal events, providing deeper insights into how shared meaning-making evolves over time. In contrast, the triangulating approach effectively identifies cumulative interaction patterns but does not account for their temporal structure. Specifically, the interleaving approach, employing epistemic network analysis, revealed statistically significant differences in discourse patterns between learners with larger and smaller variances in upper body movements during the co-design activity. These findings underscore the complementary value of the interleaving approach in analyzing multimodal interactions and offer practical implications for advancing understanding of collaborative learning processes in TEL environments.

Collaborative learning, driven by knowledge co‐construction and meaning negotiation, is a pivotal aspect of educational contexts. While gesture's importance in conveying shared meaning is recognized, its role in collaborative group settings remains understudied. This gap hinders accurate and equitable assessment and instruction, particularly for linguistically diverse students. Advancements in multimodal learning analytics, leveraging sensor technologies, offer innovative solutions for capturing and analysing body movements. This study employs these novel approaches to demonstrate how learners' machine‐detected body movements during the learning process relate to their verbal and nonverbal contributions to the co‐construction of embodied math knowledge. These findings substantiate the feasibility of utilizing learners' machine‐detected body movements as a valid indicator for inferring their engagement with the collaborative knowledge construction process. In addition, we empirically validate that these inferred different levels of learner engagement indeed impact the desired learning outcomes of the intervention. This study contributes to our scientific understanding of multimodal approaches to knowledge expression and assessment in learning, teaching, and collaboration. Practitioner notes: What is already known about this topic: Previous research emphasizes the importance of gestures as essential tools for constructing common ground and reflecting shared meaning‐making in learning and teaching contexts. The prior studies in multimodal learning analytics (MMLA) suggest that certain forms of body movements and postures can be differentiated based on the automatic detection of upper body joint locations. Empirical observations indicate that co‐thought gestures typically involve smaller hand or arms movement that are closer to the gesturer's body than co‐speech gestures used in interpersonal communication. What this paper adds: This paper fills the research gap by examining the use of gestures in collaborative learning, offering insights into how individuals contribute verbally and nonverbally to collaborative knowledge construction. This paper introduces the concept of using machine‐detected body movements as a viable proxy for inferring learners' engagement in collaborative knowledge‐building activities. Leverages sensor technologies for automatic detection of body movements, the innovative approach in this work seeks to overcome the time‐intensive and laborious process of manually coding gestures. Implications for practice and/or policy: By recognizing the potential significance of learners' body movements in indicating engagement levels with collaborative knowledge‐building activities, instructors can set up computer‐supported collaborative learning (CSCL) environments to enable capturing these movements. Given the crucial role of gestures in learning, teaching, and collaboration, educators can create more equitable formative assessment practices for linguistically diverse students by developing strategies that align with multimodal forms of knowledge expression. Research can expand beyond mathematics to explore the transferability of these findings to other subjects, helping educators create comprehensive pedagogical approaches that leverage multimodal interactions across disciplines.

In many STEM domains, instruction on foundational concepts heavily relies on visuals. Instructors often assume that students can mentally visualize concepts, but students often struggle with internal visualization skills—the ability to mentally visualize information. In order to address this issue, we developed a formal as well as an informal assessment of students’ internal visualization skills in the context of engineering instruction. To validate the assessments, we used data triangulation methods. We drew on data from two separate studies conducted in a small-scale lab experiment and in a larger-scale classroom context. Our studies demonstrate that an intelligent tutoring system with interactive visual representations can serve as an informal assessment of students’ internal visualization skills, predicting their performance on a formal assessment of these skills. Our study enriches methodological and theoretical underpinnings in educational research and practices in multiple ways: it contributes to (1) research methodologies by illustrating how multimodal triangulation can be used for test development, (2) theories of learning by offering pathways to assessing internal visualization skills that are not directly observable, and (3) instructional practices in STEM education by enabling instructors to determine when and where they should provide additional scaffoldings.

K-12 pre-service teachers (PSTs, n=16) participated in a collaborative online teacher professional development program aimed at increasing their noticing and beliefs about embodied mathematical reasoning. Before and after the intervention, PSTs assessed the quality of students’ geometric reasoning as portrayed in videos of student speech and gestures, and completed a validated survey regarding teachers’ attitudes about gesture for learning and instruction. PSTs then participated in the collaborative embodied professional learning experiences, which included playing and then designing content for a motion-capture video game for improving students’ geometric reasoning and proof production through embodied interactions. Results from multimodal discourse analysis of video-prompted interviews with PSTs, epistemic network analysis, and the surveys reveal significant improvements in PSTs’ noticing, interpreting, and appreciating of embodied forms of mathematical ways of knowing, and improved attitudes about the role of gestures in learning and teaching. Implications are offered for developing teachers’ noticing skills and formative assessment practices that include students’ nonverbal ways of expressing mathematical reasoning, and for exploring embodied design activities to influence teachers’ attitudes and beliefs about the embodied nature of student thinking and learning.

This work has been published in peer reviewed conference proceedings, please use the following reference (APA) to cite this work:Sung, H., Kim, D., Swart, M., & Nathan, M. (2023). Multimodal Behavior Analysis: Two Patterns of Collaborative Construction of Embodied Knowledge. Proceedings of the Annual Meeting of the Cognitive Science Society, 45. Retrieved from https://escholarship.org/uc/item/3nb4h58tThis study investigates how individuals collaboratively constructed shared knowledge during a group activity. The dataset was collected from group activities for pre-service teachers in professional development. Participants designed body poses and action sequences that could help their students’ mathematical conceptualization. Using k-means clustering and Principal component analysis, patterns of individuals’ contributions based on their verbal and gestural behavior identified two groups of individuals: (1) Those who contributed to the discussion by speaking and gesturing frequently (~ 25% of the participants), and (2) those who mostly listened and focused on design ideas presented by others. Furthermore, epistemic network analysis corroborated significant differences in discourse patterns between the clusters, the results of which have significant implications for collaborative embodied learning and application for teacher education and professional development.