MFA-CNN: An Emotion Recognition Network Integrating 1D–2D Convolutional Neural Network and Cross-Modal Causal Features
Open Access
- 29 October 2025
- journal article
- research article
- Published by MDPI AG in Brain Sciences
- Vol. 15 (11) , 1165
Abstract
Background/Objectives: It has become a major direction of research in affective computing to explore the brain-information-processing mechanisms based on physiological signals such as electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS). However, existing research has mostly focused on feature- and decision-level fusion, with little investigation into the causal relationship between these two modalities. Methods: In this paper, we propose a novel emotion recognition framework for the simultaneous acquisition of EEG and fNIRS signals. This framework integrates the Granger causality (GC) method and a modality–frequency attention mechanism within a convolutional neural network backbone (MFA-CNN). First, we employed GC to quantify the causal relationships between the EEG and fNIRS signals. This revealed emotional-processing mechanisms from the perspectives of neuro-electrical activity and hemodynamic interactions. Then, we designed a 1D2D-CNN framework that fuses temporal and spatial representations and introduced the MFA module to dynamically allocate weights across modalities and frequency bands. Results: Experimental results demonstrated that the proposed method outperforms strong baselines under both single-modal and multi-modal conditions, showing the effectiveness of causal features in emotion recognition. Conclusions: These findings indicate that combining GC-based cross-modal causal features with modality–frequency attention improves EEG–fNIRS-based emotion recognition and provides a more physiologically interpretable view of emotion-related brain activity.Keywords
Funding Information
- the Fundamental Research Program of Shanxi Province (No. 202403021212260)
- the Doctoral Scientific Research Starting Foundation of Taiyuan University of Science and Technology (No. 2023098)
- the Funding Awards for Outstanding Doctors Volunteering to Work in Shanxi Province (No. 20242011)
- the Science and Technology Innovation Project of Colleges and Universities in Shanxi Province (No. 2024L259)
This publication has 46 references indexed in Scilit:
- Unraveling trust in collaborative human-machine intelligence from neurophysiological perspective: a review of EEG and fNIRS featuresAdvanced Engineering Informatics, 2025
- A review of hybrid EEG-based multimodal human–computer interfaces using deep learning: applications, advances, and challengesBiomedical Engineering Letters, 2025
- Motor Imagery EEG signals classification using a Transformer-GCN approachApplied Soft Computing, 2024
- Deep learning-based EEG emotion recognition: a comprehensive reviewNeural Computing and Applications, 2024
- Early EEG and NIRS measurements in preterm babies: a systematic reviewEuropean Journal of Pediatrics, 2024
- Motor imagery EEG task recognition using a nonlinear Granger causality feature extraction and an improved Salp swarm feature selectionBiomedical Signal Processing and Control, 2023
- Emotion recognition using effective connectivity and pre-trained convolutional neural networks in EEG signalsCognitive Neurodynamics, 2022
- A novel classification framework using multiple bandwidth method with optimized CNN for brain–computer interfaces with EEG-fNIRS signalsNeural Computing and Applications, 2021
- A systematic review on hybrid EEG/fNIRS in brain-computer interfaceBiomedical Signal Processing and Control, 2021
- Convolutional Neural Network for Hybrid fNIRS-EEG Mental Workload Classification2019