Abstract
Background/Objectives: It has become a major direction of research in affective computing to explore the brain-information-processing mechanisms based on physiological signals such as electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS). However, existing research has mostly focused on feature- and decision-level fusion, with little investigation into the causal relationship between these two modalities. Methods: In this paper, we propose a novel emotion recognition framework for the simultaneous acquisition of EEG and fNIRS signals. This framework integrates the Granger causality (GC) method and a modality–frequency attention mechanism within a convolutional neural network backbone (MFA-CNN). First, we employed GC to quantify the causal relationships between the EEG and fNIRS signals. This revealed emotional-processing mechanisms from the perspectives of neuro-electrical activity and hemodynamic interactions. Then, we designed a 1D2D-CNN framework that fuses temporal and spatial representations and introduced the MFA module to dynamically allocate weights across modalities and frequency bands. Results: Experimental results demonstrated that the proposed method outperforms strong baselines under both single-modal and multi-modal conditions, showing the effectiveness of causal features in emotion recognition. Conclusions: These findings indicate that combining GC-based cross-modal causal features with modality–frequency attention improves EEG–fNIRS-based emotion recognition and provides a more physiologically interpretable view of emotion-related brain activity.
Funding Information
  • the Fundamental Research Program of Shanxi Province (No. 202403021212260)
  • the Doctoral Scientific Research Starting Foundation of Taiyuan University of Science and Technology (No. 2023098)
  • the Funding Awards for Outstanding Doctors Volunteering to Work in Shanxi Province (No. 20242011)
  • the Science and Technology Innovation Project of Colleges and Universities in Shanxi Province (No. 2024L259)

This publication has 46 references indexed in Scilit: