Abstract
Multimodal emotion recognition based on complementary physiological signals, such as an electroencephalogram (EEG) and eye movement, can effectively reflect human emotional states, showing great potential in fields like rehabilitation monitoring and driving safety management. However, existing methods usually directly map multimodal features into a unified feature space for information fusion, which not only fails to eliminate the heterogeneity between different modalities but also ignores the impact of the modal contribution on the fusion results. To address these limitations, this paper proposes a cross-modal alignment and dynamic fusion (CMA-DF) learning framework to improve the performance of multimodal emotion recognition. Specifically, we first adopt a two-branch feature extractor to respectively extract robust feature representations of EEG signals and eye movement signals. Secondly, a cross-modal alignment network is designed, which reduces distribution differences through collaborative modeling of intermodal interaction relationships and alignment strategies, thus providing a consistent basis for subsequent fusion. Subsequently, a dynamic fusion strategy is introduced to efficiently integrate complementary information from multiple modalities while adaptively balancing the contribution weights of each modality, thereby enhancing the representational capability of the fused features. Finally, experimental results in the SEED and SEED-V datasets show that the proposed method achieves recognition accuracies of 96.69% and 88.03%, respectively. Comparative experiments fully verify its superiority in multimodal feature alignment, fusion, and emotion recognition performance.
Funding Information
  • Zhao Lv (No. 2024AH050054)
  • Zhao Lv (No. 2208085J05)
  • Zhao Lv (Nos. 62476004, 62501007)

This publication has 53 references indexed in Scilit: