Emotion Recognition Using a Siamese Model and a Late Fusion-Based Multimodal Method in the WESAD Dataset with Hardware Accelerators
Author:
Open Access
- 13 February 2025
- journal article
- research article
- Published by MDPI AG in Electronics
- Vol. 14 (4) , 723
Abstract
Emotion recognition, which includes stress analysis, is a critical research topic. In particular, emotion recognition using wearable applications that offer rapid feedback can significantly assist in emotion regulation. A key dataset for this purpose is the wearable stress and affect detection (WESAD) dataset, which is well-suited for wearable environments. This study aims to evaluate the accuracy and processing time of the proposed neural network using the WESAD dataset. However, data collected in wearable environments often have limitations in terms of their accuracy and channel count, making classification challenging. To address this issue, this study utilizes only wrist-measured blood volume pulse (BVP) data measured by photo-plethysmography (PPG) and electro-dermal activity (EDA) data, classifying them through a late fusion-based multimodal method. For accurate classification, the BVP data are preprocessed using the empirical mode decomposition (EMD) method, and a Siamese model is employed for learning on small datasets, with an additional structure of dense layers for detailed classification. The Siamese network generates a similarity score, which is then fed into a dense layer to classify multiple classes. Meta-classes are used to compute pairwise similarities, ultimately classifying the data into one of four emotional states: neutral, stress, amusement, and meditation. The proposed multitasking model simultaneously trains the Siamese network and multi-classifier, achieving an accuracy of 99.8% in classifying four emotion states in the WESAD dataset. Moreover, the model is designed to operate with minimal hardware resources, making it easy to implement using hardware accelerators. This study demonstrates the applicability of neural network-based wearable systems in the field of emotion recognition.Keywords
Funding Information
- Chosun University (2024)
This publication has 41 references indexed in Scilit:
- Ensemble machine learning model trained on a new synthesized dataset generalizes well for stress prediction using wearable devicesJournal of Biomedical Informatics, 2023
- Real-time mental stress detection technique using neural networks towards a wearable health monitorMeasurement Science and Technology, 2022
- Deep convolution network based emotion analysis towards mental health careNeurocomputing, 2020
- Emotion Recognition from Multimodal Physiological Signals for Emotion Aware Healthcare SystemsJournal of Medical and Biological Engineering, 2020
- Towards an automated design flow for memristor based VLSI circuitsIntegration, 2020
- Emotion Recognition from Physiological Signal Analysis: A ReviewElectronic Notes in Theoretical Computer Science, 2019
- Facial emotion recognition in Parkinson's disease: A review and new hypothesesMovement Disorders, 2018
- An accurate emotion recognition system using ECG and GSR signals and matching pursuit methodBiomedical Journal, 2017
- Emotion recognition using facial expressionsProcedia Computer Science, 2017
- RETRACTED ARTICLE: Automatic detection of sleep spindles with the use of STFT, EMD and DWT methodsNeural Computing and Applications, 2016