Hierarchical Hypercomplex Network for Multimodal Emotion Recognition
Authors:
- 22 September 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-6
Abstract
Emotion recognition is relevant in various domains, ranging from healthcare to human-computer interaction. Physiological signals, being beyond voluntary control, offer reliable information for this purpose, unlike speech and facial expressions which can be controlled at will. They reflect genuine emotional responses, devoid of conscious manipulation, thereby enhancing the credibility of emotion recognition systems. Nonetheless, multimodal emotion recognition with deep learning models remains a relatively unexplored field. In this paper, we introduce a fully hypercomplex network with a hierarchical learning structure to fully capture correlations. Specifically, at the encoder level, the model learns intra- modal relations among the different channels of each input signal. Then, a hypercomplex fusion module learns inter-modal relations among the embeddings of the different modalities. The main novelty is in exploiting intra-modal relations by endowing the encoders with parameterized hyper-complex convolutions (PHCs) that thanks to hypercomplex algebra can capture inter-channel interactions within single modalities. Instead, the fusion module comprises parameterized hypercomplex multiplications (PHMs) that can model inter-modal correlations. The proposed architecture surpasses state-of-the-art models on the MAHNOB-HCI dataset for emotion recognition, specifically in classifying valence and arousal from electroencephalograms (EEGs) and peripheral physiological signals. The code of this study is available at https://github.com/ispamm/MHyEEG.Keywords
All Related Versions
This publication has 20 references indexed in Scilit:
- Demystifying the Hypercomplex: Inductive biases in hypercomplex deep learningIEEE Signal Processing Magazine, 2024
- Hypercomplex Multimodal Emotion Recognition from EEG and Peripheral Physiological Signals2023
- An attention-based hybrid deep learning model for EEG emotion recognitionSignal, Image and Video Processing, 2022
- Multimodal emotion recognition based on manifold learning and convolution neural networkMultimedia Tools and Applications, 2022
- Feature Reconstruction Based Channel Selection for Emotion Recognition Using EEG2021
- Hierarchical Attention-Based Temporal Convolutional Networks for Eeg-Based Emotion Recognition2021
- Emotion Recognition From EEG Signal Focusing on Deep Learning and Shallow Learning TechniquesIEEE Access, 2021
- An Efficient LSTM Network for Emotion Recognition From Multichannel EEG SignalsIEEE Transactions on Affective Computing, 2020
- Real-Time Movie-Induced Discrete Emotion Recognition from EEG SignalsIEEE Transactions on Affective Computing, 2017
- A Multimodal Database for Affect Recognition and Implicit TaggingIEEE Transactions on Affective Computing, 2011