Multimodal data stream classification and prediction of e-learner’s emotional states
Author:
- Arijit NandiANArijit Nandi
Open Access
- 17 February 2026
- thesis
- Published by Universitat Politècnica de Catalunya
Abstract
(English) Emotions and emotional intelligence are crucial for students' success both in traditional learning environments (face-to-face classroom), online education (or E-Learning/distance learning), and in more recently developed technological environments such as augmented workspaces (i.e., adaptive training systems based on augmented or virtual reality). Many researchers have examined the role of emotions in improving and optimizing the outcomes of those learning scenarios. Several approaches (ranging from machine learning/ deep learning models) have also been proposed to use one or more data modalities for emotion classification. However, those ML/DL models so far are suitable for offline mode, where data for emotion classification is stored and can be accessed infinitely. In addition, the recent technical developments and emerging Internet of Things (IoT) technologies have added a new dimension to those learning environments, with numerous feedback instruments (such as webcams, fit-bands, and so on) for engagement with learners. This new layer offers immense quantities of data with volume, diversity, and velocity, referred to as multi-modal data streams, as well as additional complexities in real-time data classification. Thus, the core scientific challenge involves systematically designing, developing, and evaluating a system for real-time emotion classification of learners using multi-modal data streams in those learning scenarios. Initially, this thesis investigates and answers the fundamental research questions by concentrating on the construction of a real- time emotion classification system employing multi-modal data streams, with a major focus on physiological signals (such as Electroencephalogram( EEG ), Electrocardiogram ( ECG), Electrodermal Activity ( EDA) and Respiratory Belt (RB)). Then, to address the fundamental questions raised in this thesis, we have devised a series of distinct methodologies, including Real-time Emotion Classification System (RECS ), Real-time Multi-modal Emotion classification System (ReMECS), and Reward Penalty Based Weighted Ensemble ( RPWE). Furthermore, the thesis extends the ReMECS approach to handle decentralized multi- modal data stream processing for real-time emotion classification with users privacy protection, called Federated Learning based ReMECS ( Fed-ReMECS). In addition, the solution of easy deployment of the federated learning-based approach (Docker- enabled Federated Learning ( DFL)) across distributed devices has also been proposed. All these proposed approaches performance is rigorously evaluated using two benchmark datasets, A Database for Emotion Analysis using Physiological Signals ( DEAP) and A Dataset for Affect, Personality and Mood Research on Individuals and Groups ( AMIGOS), demonstrating the superiority over existing state-of-the-art models. Note that these datasets are accessible after signing the End User License Agreement (EULA). As a result, this thesis contributes fundamentally to the core knowledge base in real-time emotion classification using multi-modal data streaming as a necessary prerequisite to precise and robust emotional interaction in different environments in the context of E-Learning. (Català) Les emocions i la intel·ligència emocional són crucials per a l'èxit dels estudiants tant en entorns d'aprenentatge tradicionals (aules presencials), educació en línia (o e-learning / aprenentatge a distància) com en entorns tecnològics desenvolupats més recentment, com en espais de treball augmentats (és a dir, sistemes d'entrenament adaptatiu basats en realitat augmentada o virtual). Molts investigadors han examinat el paper de les emocions en la millora i l'optimització dels resultats d'aquests escenaris d'aprenentatge. També s'han proposat diversos enfocaments (que van des de models d'aprenentatge automàtic/aprenentatge profund) per utilitzar una o més modalitats de dades per a la classificació de les emocions. No obstant, fins ara aquests models ML/DL són adequats per al mode offline, on les dades per a la classificació d'emocions s'emmagatzemen s hi pot accedir infinitament. A més, els recents avenços tècnics i les tecnologies emergents d'Internet de les coses (IoT) han afegit una nova dimensió a aquests entorns d'aprenentatge, amb nombrosos instruments de retroalimentació (com càmeres web, polseres esportives, i així successivament) per a la interacció amb els alumnes. Aquesta nova capacitat ofereix grans quantitats de dades amb volum, diversitat i velocitat, conegudes com a fluxos de dades multimodals, així com a complements addicionals en la classificació de dades en temps real. Per tant, el desafiament científic central implica dissenyar, desenvolupar i avaluar sistemàticament un sistema per a la classificació de les emocions dels estudiants en el temps real utilitzant fluxos de dades multimodals en aquests escenaris d'aprenentatge. Inicialment, aquesta tesi investiga i respon a les preguntes d'investigació fonamentals concentrant-se en la construcció d'un sistema de classificació d'emocions en temps real que fa servir fluxos de dades multimodals, amb un enfocament principal en senyals fisiològics (com a electroencefalograma (EEG), electrocardiograma (ECG), activitat electrodèrmica (EDA) i cinturó respiratori (RB)). Després, per abordar les preguntes fonamentals plantejades en aquesta tesi, hem ideat una sèrie de metodologies diferents, que inclouen un sistema de classificació d'emocions en temps real (RECS), un sistema de classificació d'emocions multimodal en temps real (ReMECS) i un conjunt ponderat basat en penalització de recompensa (RPWE). A més, la tesi amplia l'enfocament ReMECS per manejar el processament de flux de dades multimodal descentralitzat per a la classificació...Keywords
This publication has 135 references indexed in Scilit:
- Federated learning inspired privacy sensitive emotion recognition based on multi-modal physiological sensorsCluster Computing, 2023
- A Docker-based federated learning framework design and deployment for multi-modal data stream classificationComputing, 2023
- Emotional outcomes of e-learning adoption during compulsory online educationEducation and Information Technologies, 2022
- Emotion recognition using support vector machine and one-dimensional convolutional neural networkMultimedia Tools and Applications, 2021
- Data stream analysis: Foundations, major tasks and toolsWIREs Data Mining and Knowledge Discovery, 2021
- EEG Emotion Recognition Based on Graph Regularized Sparse Linear RegressionNeural Processing Letters, 2018
- Adaptive random forests for evolving data stream classificationMachine Learning, 2017
- Advances in data stream miningWIREs Data Mining and Knowledge Discovery, 2011
- Classes That Click: Fast, Rich Feedback to Enhance Student Learning and SatisfactionJournal of Engineering Education, 2010
- The Supervised Learning No-Free-Lunch Theorems2002