All rivers run into the sea: Unified Modality Brain-like Emotional Central Mechanism
Authors:
- Xinji MaiXMXinji Mai
- Junxiong LinJLJunxiong Lin
- Haoran WangHWHaoran Wang
- Zeng TaoZTZeng Tao
- Yan WangYWYan Wang
- Shaoqi YanSYShaoqi Yan
- Xuan TongXTXuan Tong
- Jiawen YuJYJiawen Yu
- Boyang WangBWBoyang Wang
- Ziheng ZhouZZZiheng Zhou
- Qing ZhaoQZQing Zhao
- Shuyong GaoSGShuyong Gao
Preprint
Abstract
In the field of affective computing, fully leveraging information from a variety of sensory modalities is essential for the comprehensive understanding and processing of human emotions. Inspired by the process through which the human brain handles emotions and the theory of cross-modal plasticity, we propose UMBEnet, a brain-like unified modal affective processing network. The primary design of UMBEnet includes a Dual-Stream (DS) structure that fuses inherent prompts with a Prompt Pool and a Sparse Feature Fusion (SFF) module. The design of the Prompt Pool is aimed at integrating information from different modalities, while inherent prompts are intended to enhance the system's predictive guidance capabilities and effectively manage knowledge related to emotion classification. Moreover, considering the sparsity of effective information across different modalities, the SSF module aims to make full use of all available sensory data through the sparse integration of modality fusion prompts and inherent prompts, maintaining high adaptability and sensitivity to complex emotional states. Extensive experiments on the largest benchmark datasets in the Dynamic Facial Expression Recognition (DFER) field, including DFEW, FERV39k, and MAFW, have proven that UMBEnet consistently outperforms the current state-of-the-art methods. Notably, in scenarios of Modality Missingness and multimodal contexts, UMBEnet significantly surpasses the leading current methods, demonstrating outstanding performance and adaptability in tasks that involve complex emotional understanding with rich multimodal information.Keywords
All Related Versions
This publication has 20 references indexed in Scilit:
- Deep learning-based multimodal emotion recognition from audio, visual, and text modalities: A systematic review of recent advancements and future prospectsExpert Systems with Applications, 2023
- Emotion recognition from unimodal to multimodal analysis: A reviewInformation Fusion, 2023
- Mapping thalamic innervation to individual L2/3 pyramidal neurons and modeling their ‘readout’ of visual inputNature Neuroscience, 2023
- Multimodal Video Sentiment Analysis Using Deep Learning Approaches, a SurveyInformation Fusion, 2021
- Cross-modal plasticity in developmental and age-related hearing loss: Clinical implicationsHearing Research, 2017
- The Functional Anatomy of Time: What and When in the BrainTrends in Cognitive Sciences, 2016
- Fusing audio, visual and textual clues for sentiment analysis from multimodal contentNeurocomputing, 2016
- The Role of Text Pre-processing in Sentiment AnalysisProcedia Computer Science, 2013
- Brain anatomical networks in early human brain developmentNeuroImage, 2011
- Critical Period for Cross-Modal Plasticity in Blind Humans: A Functional MRI StudyNeuroImage, 2002