Adaptive information fusion network for multi‐modal personality recognition
- 1 May 2024
- journal article
- research article
- Published by Wiley in Computer Animation and Virtual Worlds
- Vol. 35 (3)
Abstract
Personality recognition is of great significance in deepening the understanding of social relations. While personality recognition methods have made significant strides in recent years, the challenge of heterogeneity between modalities during feature fusion still needs to be solved. This paper introduces an adaptive multi‐modal information fusion network (AMIF‐Net) capable of concurrently processing video, audio, and text data. First, utilizing the AMIF‐Net encoder, we process the extracted audio and video features separately, effectively capturing long‐term data relationships. Then, adding adaptive elements in the fusion network can alleviate the problem of heterogeneity between modes. Lastly, we concatenate audio‐video and text features into a regression network to obtain Big Five personality trait scores. Furthermore, we introduce a novel loss function to address the problem of training inaccuracies, taking advantage of its unique property of exhibiting a peak at the critical mean. Our tests on the ChaLearn First Impressions V2 multi‐modal dataset show partial performance surpassing state‐of‐the‐art networks.Keywords
Funding Information
- National Natural Science Foundation of China (62072020)
- National Science and Technology Major Project (2022ZD0119502)
- Natural Science Foundation of Beijing Municipality (L222052)
- Shandong University of Science and Technology (BJ20231201)
This publication has 24 references indexed in Scilit:
- Combining machine learning algorithms for personality trait predictionEgyptian Informatics Journal, 2024
- TMSDNet: Transformer with multi‐scale dense network for single and multi‐view 3D reconstructionComputer Animation and Virtual Worlds, 2023
- DesPrompt: Personality-descriptive prompt tuning for few-shot personality recognitionInformation Processing & Management, 2023
- RAIF: A deep learning‐based architecture for multi‐modal aesthetic biometric systemComputer Animation and Virtual Worlds, 2023
- SCANET: Improving multimodal representation and fusion with sparse‐ and cross‐attention for multimodal sentiment analysisComputer Animation and Virtual Worlds, 2022
- Multi-modal Transformer for Video Retrieval2020
- CR-Net: A Deep Classification-Regression Network for Multimodal Apparent Personality AnalysisInternational Journal of Computer Vision, 2020
- Physiognomy: Personality traits prediction by learningInternational Journal of Automation and Computing, 2017
- Deep Impression: Audiovisual Deep Residual Networks for Multimodal Apparent Personality Trait Recognition2016
- ChaLearn LAP 2016: First Round Challenge on First Impressions - Dataset and Results2016