Correlation-Driven Multi-Modality Graph Decomposition for Cross-Subject Emotion Recognition
Open Access
- 28 October 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 2272-2281
Abstract
Multi-modality physiological signal-based emotion recognition has attracted increasing attention as its capacity to capture human affective states comprehensively. Due to multi-modality heterogeneity and cross-subject divergence, practical applications struggle with generalizing models across individuals. Effectively addressing both issues requires mitigating the gap between multimodal signals while acquiring generalizable representations across subjects. However, existing approaches often handle these dual challenges separately, resulting in suboptimal generalization. This study introduces a novel framework, termed Correlation-Driven Multi-Modality Graph Decomposition (CMMGD). The proposed CMMGD initially captures adaptive cross-modal correlations. It connects each unimodal graph to a multimodal mixed graph. To simultaneously address the dual challenges, it incorporates a correlation-driven graph decomposition module that decomposes the mixed graph into concordant and discrepant subgraphs based on the correlations. The decomposed concordant subgraph encompasses consistently activated features across modalities and subjects during emotion elicitation, unveiling a generalizable subspace. Additionally, we design a Multi-Modality Graph Regularized Transformer (MGRT) backbone specifically tailored for multimodal physiological signals. The MGRT can alleviate the over-smoothing issue and mitigate over-reliance on any single modality. Extensive experiments demonstrate that CMMGD outperforms the state-of-the-art methods by 1.79% and 2.65% on DEAP and MAHNOB-HCI datasets, respectively, under the leave-one-subject-out cross-validation strategy.Keywords
Funding Information
- Youth Innovation Promotion Association CAS
- Hunan Provincial Natural Science Foundation of China (2023JJ70034, 2023JJ70008)
- National Key Research and Development Plan of China (2023YFC3604802)
- science and technology innovation Program of Hunan Province (2022RC4006, 2023WK2005)
This publication has 46 references indexed in Scilit:
- Hierarchical multimodal-fusion of physiological signals for emotion recognition with scenario adaption and contrastive alignmentInformation Fusion, 2023
- On the Versatile Uses of Partial Distance Correlation in Deep Learning2022
- Affect recognition from scalp-EEG using channel-wise encoder networks coupled with geometric deep learning and multi-channel feature fusionKnowledge-Based Systems, 2022
- EEG emotion recognition using fusion model of graph convolutional neural networks and LSTMApplied Soft Computing, 2021
- Locally robust EEG feature selection for individual-independent emotion recognitionExpert Systems with Applications, 2020
- Emotion recognition using multi-modal data and machine learning techniques: A tutorial and reviewInformation Fusion, 2020
- Exploring temporal representations by leveraging attention-based bidirectional LSTM-RNNs for multi-modal emotion recognitionInformation Processing & Management, 2020
- Multimodal Emotion Recognition Using Deep Neural Networks2017
- Emotional state classification from EEG data using machine learning approachNeurocomputing, 2014
- Feature extraction from EEGs associated with emotionsArtificial Life and Robotics, 1997