Abstract
Multi-modal personality traits recognition is crucial for advancing human-computer interaction, yet current methods struggle with extracting distinguishable features, mitigating inter-modal noise, and aligning heterogeneous modalities, thereby limiting accuracy. To address these challenges, we propose an emotion-enhanced multi-modal fusion model leveraging supervised contrastive learning for personality traits recognition. Specifically, we use supervised contrastive learning to learn highly distinguishable representations of modalities. Capitalizing on the intrinsic link between emotions and personality, we introduce a novel emotion-enhanced multi-modal fusion mechanism. This mechanism strategically guides the fusion process, reducing inter-modal noise and sharpening feature alignment across modalities. Each mechanism provides flexible designs for diverse personality traits recognition scenarios, optimizing both feature extraction and fusion for superior accuracy. In addition, we explore the interrelations among cultural contexts, emotions, and personality traits. Extensive experiments on three diverse datasets validate the superior performance and generalization of our proposed method.
Funding Information
  • Japan Society for the Promotion of Science London (JP20H04491)
  • Japan Society for the Promotion of Science London (JP24K02974)