Multi-Domain Encoding of Spatiotemporal Dynamics in EEG for Emotion Recognition
- 27 December 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal of Biomedical and Health Informatics
- Vol. 27 (3) , 1342-1353
Abstract
The common goal of the studies is to map any emotional states encoded from electroencephalogram (EEG) into 2-dimensional arousal-valance scores. It is still challenging due to each emotion having its specific spatial structure and dynamic dependence over the distinct time segments among EEG signals. This paper aims to model human dynamic emotional behavior by considering the location connectivity and context dependency of brain electrodes. Thus, we designed a hybrid EEG modeling method that mainly adopts the attention mechanism, combining a multi-domain spatial transformer (MST) module and a dynamic temporal transformer (DTT) module, named MSDTTs. Specifically, the MST module extracts single-domain and cross-domain features from different brain regions and fuses them into multi-domain spatial features. Meanwhile, the temporal dynamic excitation (TDE) is inserted into the multi-head convolutional transformer to form the DTT module. These two blocks work together to activate and extract the emotion-related dynamic temporal features within the DTT module. Furthermore, we place the convolutional mapping into the transformer structure to mine the static context features among the keyframes. Overall results show that high classification accuracy of 98.91%/0.14% was obtained by the $\beta$ frequency band of the DEAP dataset, and 97.52%/0.12% and 96.70%/0.26% were obtained by the $\gamma$ frequency band of SEED and SEED-IV datasets. Empirical experiments indicate that our proposed method can achieve remarkable results in comparison with state-of-the-art algorithms.Keywords
Funding Information
- National Natural Science Foundation of China (61972064, 61772252)
- Liaoning Revitalization Talents Program (XLYC1806006)
- Fundamental Research Funds for the Central Universities (DU19RC(3)012)
This publication has 44 references indexed in Scilit:
- Comparison of hemispheric asymmetry measurements for emotional recordings from controlsNeural Computing and Applications, 2017
- Relationships between the resting-state network and the P3: Evidence from a scalp EEG studyScientific Reports, 2015
- Investigating Critical Frequency Bands and Channels for EEG-Based Emotion Recognition with Deep Neural NetworksIEEE Transactions on Autonomous Mental Development, 2015
- Identifying Emotions on the Basis of Neural ActivationPLOS ONE, 2013
- Multimodal Emotion Recognition in Response to VideosIEEE Transactions on Affective Computing, 2011
- DEAP: A Database for Emotion Analysis ;Using Physiological SignalsIEEE Transactions on Affective Computing, 2011
- The dynamic emotion recognition system based on functional connectivity of brain regions2010
- EEG-Based Emotion Recognition in Music ListeningIEEE Transactions on Biomedical Engineering, 2010
- Electroencephalographic spectral asymmetry index for detection of depressionMedical & Biological Engineering & Computing, 2009
- Bad is Stronger than GoodReview of General Psychology, 2001