Evaluating Vision Language Models in Detecting Learning Engagement
Authors:
- 9 December 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 496-502
Abstract
With the advancement of both computer vision and natural language processing, there is growing interest in incorporating Vision Language Models (VLMs) into the classroom to empower students and educators. Despite the VLMs’ sophisticated abilities in context-aware emotion recognition, their effectiveness in detecting classroom-specific emotions, e.g., engagement, distraction, and absent-mindedness, remains underexplored. As such, this paper aims to investigate the capabilities of two state-of-the-art VLMs in this domain through an empirical study, focusing on two research questions: 1) Is learning engagement detection more challenging for VLMs compared to conventional emotion detection? 2) What are the key difficulties faced by VLMs in processing learning engagement detection tasks? To address these questions, we perform a series of evaluation experiments by utilizing a classroom behavior detection dataset and an emotion recognition dataset. We conclude that VLMs that perform well on basic emotion recognition struggle with in-context engagement detection, due to the nuanced and context-dependent nature of the task. Specifically, experiments show that VLMs have difficulty distinguishing engaged and distracted classroom behavior, e.g., reading versus bowing the head. It suggests that VLMs still have significant room for improvement in engagement analysis. This issue can potentially be addressed by incorporating more classroom-specific training data or commonsense reasoning frameworks.Keywords
This publication has 48 references indexed in Scilit:
- Understanding Natural Language Understanding2025
- SenticNet 8: Fusing Emotion AI and Commonsense AI for Interpretable, Trustworthy, and Explainable Affective Computing2024
- Neurosymbolic AI for Personalized Sentiment Analysis2024
- Hierarchical multimodal-fusion of physiological signals for emotion recognition with scenario adaption and contrastive alignmentInformation Fusion, 2023
- Semi-supervised cross-lingual speech emotion recognitionExpert Systems with Applications, 2023
- Finding hate speech with auxiliary emotion detection from self-training multi-label learning perspectiveInformation Fusion, 2023
- Attention-based multimodal contextual fusion for sentiment and emotion classification using bidirectional LSTMMultimedia Tools and Applications, 2021
- Smart education literature: A theoretical analysisEducation and Information Technologies, 2020
- ARBEE: Towards Automated Recognition of Bodily Expression of Emotion in the WildInternational Journal of Computer Vision, 2019
- Consensus vote models for detecting and filtering neutrality in sentiment analysisInformation Fusion, 2018