E²MP: Emotion-Enhanced Modality Fusion With Contrastive Learning for Multi-Modal Personality Traits Recognition
Open Access
- 31 March 2026
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 14, 50516-50536
Abstract
Multi-modal personality traits recognition is crucial for advancing human-computer interaction, yet current methods struggle with extracting distinguishable features, mitigating inter-modal noise, and aligning heterogeneous modalities, thereby limiting accuracy. To address these challenges, we propose an emotion-enhanced multi-modal fusion model leveraging supervised contrastive learning for personality traits recognition. Specifically, we use supervised contrastive learning to learn highly distinguishable representations of modalities. Capitalizing on the intrinsic link between emotions and personality, we introduce a novel emotion-enhanced multi-modal fusion mechanism. This mechanism strategically guides the fusion process, reducing inter-modal noise and sharpening feature alignment across modalities. Each mechanism provides flexible designs for diverse personality traits recognition scenarios, optimizing both feature extraction and fusion for superior accuracy. In addition, we explore the interrelations among cultural contexts, emotions, and personality traits. Extensive experiments on three diverse datasets validate the superior performance and generalization of our proposed method.Keywords
Funding Information
- Japan Society for the Promotion of Science London (JP20H04491)
- Japan Society for the Promotion of Science London (JP24K02974)
This publication has 129 references indexed in Scilit:
- Enhancing interpretability in video-based personality trait recognition using SHAP analysisMultimedia Systems, 2025
- A deep multimodal fusion method for personality traits predictionMultimedia Tools and Applications, 2024
- YOLO5Face: Why Reinventing a Face Detector2023
- The AI4Media Project: Use of Next-Generation Artificial Intelligence Technologies for Media Sector Applications2021
- CR-Net: A Deep Classification-Regression Network for Multimodal Apparent Personality AnalysisInternational Journal of Computer Vision, 2020
- Combining Deep Facial and Ambient Features for First Impression Estimation2016
- Deep Impression: Audiovisual Deep Residual Networks for Multimodal Apparent Personality Trait Recognition2016
- All smiles perceived equally: Facial expressions trump target characteristics in impression formationMotivation and Emotion, 2016
- Common Sense Knowledge Based Personality Recognition from Text2013
- Predicting the Big Two of Affect from the Big Five of PersonalityJournal of Research in Personality, 2001