Commonalities and variations in emotion representation across modalities and brain regions
Authors:
Open Access
- 9 September 2024
- journal article
- research article
- Published by Springer Nature in Scientific Reports
- Vol. 14 (1) , 1-13
Abstract
Humans express emotions through various modalities such as facial expressions and natural language. However, the relationships between emotions expressed through different modalities and their correlations with neural activities remain uncertain. Here, we aimed to unveil some of these uncertainties by investigating the similarity of emotion representations across modalities and brain regions. First, we represented various emotion categories as multi-dimensional vectors derived from visual (face), linguistic, and visio-linguistic data, and used representational similarity analysis to compare these modalities. Second, we examined the linear transferability of emotion representation from other modalities to the visual modality. Third, we compared the representational structure derived in the first step with those from brain activities across 360 regions. Our findings revealed that emotion representations share commonalities across modalities with modality-type dependent variations, and they can be linearly mapped from other modalities to the visual modality. Additionally, emotion representations in uni-modalities showed relatively higher similarity with specific brain regions, while multi-modal emotion representation was most similar to representations across the entire brain region. These findings suggest that emotional experiences are represented differently across various brain regions with varying degrees of similarity to different modality types, and that they may be multi-modally conveyable in visual and linguistic domains.Keywords
All Related Versions
Funding Information
- Japan Society for the Promotion of Science (KAKENHI (23K16985))
- Japan Science and Technology Agency (Moonshot Research & Development Program grant JPMJMS2012)
- The National Institute of Information and Communications Technology (NICT 22301)
This publication has 42 references indexed in Scilit:
- Emotion unfolded by motion: a role for parietal lobe in decoding dynamic facial expressionsSocial Cognitive and Affective Neuroscience, 2012
- Mapping discrete and dimensional emotions onto the brain: controversies and consensusTrends in Cognitive Sciences, 2012
- Supramodal Representation of EmotionsThe Journal of Neuroscience, 2011
- Supramodal Representations of Perceived Emotions in the Human BrainThe Journal of Neuroscience, 2010
- Representational similarity analysis – connecting the branches of systems neuroscienceFrontiers in Systems Neuroscience, 2008
- Language as context for the perception of emotionTrends in Cognitive Sciences, 2007
- Language and the perception of emotion.Emotion, 2006
- On the universality and cultural specificity of emotion recognition: A meta-analysis.Psychological Bulletin, 2002
- Facial expression and emotion.American Psychologist, 1993
- Pan-Cultural Elements in Facial Displays of EmotionScience, 1969