CMA-DF: A physiological signal fusion method based on cross-modal alignment and dynamic fusion
- 9 April 2026
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Consumer Electronics
- Vol. PP (00983063) , 1
Abstract
Multimodal emotion recognition based on complementary physiological signals, such as an electroencephalogram (EEG) and eye movement, can effectively reflect human emotional states, showing great potential in fields like rehabilitation monitoring and driving safety management. However, existing methods usually directly map multimodal features into a unified feature space for information fusion, which not only fails to eliminate the heterogeneity between different modalities but also ignores the impact of the modal contribution on the fusion results. To address these limitations, this paper proposes a cross-modal alignment and dynamic fusion (CMA-DF) learning framework to improve the performance of multimodal emotion recognition. Specifically, we first adopt a two-branch feature extractor to respectively extract robust feature representations of EEG signals and eye movement signals. Secondly, a cross-modal alignment network is designed, which reduces distribution differences through collaborative modeling of intermodal interaction relationships and alignment strategies, thus providing a consistent basis for subsequent fusion. Subsequently, a dynamic fusion strategy is introduced to efficiently integrate complementary information from multiple modalities while adaptively balancing the contribution weights of each modality, thereby enhancing the representational capability of the fused features. Finally, experimental results in the SEED and SEED-V datasets show that the proposed method achieves recognition accuracies of 96.69% and 88.03%, respectively. Comparative experiments fully verify its superiority in multimodal feature alignment, fusion, and emotion recognition performance.Keywords
Funding Information
- Zhao Lv (No. 2024AH050054)
- Zhao Lv (No. 2208085J05)
- Zhao Lv (Nos. 62476004, 62501007)
This publication has 53 references indexed in Scilit:
- DTCA: Dual-Branch Transformer with Cross-Attention for EEG and Eye Movement Data Fusion2024
- Emotion recognition from unimodal to multimodal analysis: A reviewInformation Fusion, 2023
- Emotion recognition using heterogeneous convolutional neural networks combined with multimodal factorized bilinear poolingBiomedical Signal Processing and Control, 2022
- Multimodal emotion recognition based on manifold learning and convolution neural networkMultimedia Tools and Applications, 2022
- Classification of Individual’s discrete emotions reflected in facial microexpressions using electroencephalogram and facial electromyogramExpert Systems with Applications, 2022
- EEG-based emotion recognition using 4D convolutional recurrent neural networkCognitive Neurodynamics, 2020
- Emotion Recognition Using Multimodal Deep Learning2016
- Multi-modal emotion analysis from facial expressions and electroencephalogramComputer Vision and Image Understanding, 2016
- Personality, emotion, and individual differences in physiological responsesBiological Psychology, 2010
- Machine learning for real-time single-trial EEG-analysis: From brain–computer interfacing to mental state monitoringJournal of Neuroscience Methods, 2008