Fangyu Zhao
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Research Interests
Fangyu Zhao has a total of 6 co-authors. Here are the top co-authors by number of shared publications:
- Fujiang Li
College of Electronic Information Engineering, Taiyuan University of Technology, Jinzhong 030600, China
1 shared publication - Huizhi Liang
School of Computing, Newcastle University, Newcastle NE1 7RU, UK
1 shared publication - Liangqi Li
College of Electronic Information Engineering, Taiyuan University of Technology, Jinzhong 030600, China
1 shared publication - Shufei Duan
College of Computer Science and Technology, Shanxi University of Electronic Science and Technology, Linfen 041000, China
1 shared publication - Ting Zhu
School of Computing, Newcastle University, Newcastle NE1 7RU, UK
1 shared publication - Wenjie Zhang
College of Electronic Information Engineering, Taiyuan University of Technology, Jinzhong 030600, China
1 shared publication
Latest Publications
- Open Access
- 2 April 2026
- Published by MDPI AG in Multimodal Technologies and Interaction
- Vol. 10 (4) , 38
PAD-Guided Multimodal Hybrid Contrastive Emotion Recognition upon STEM-E2VA Dataset
There are still challenges in speech emotion recognition, as the representation capability of single-modal information is limited, there are difficulties in capturing continuous emotional transitions in discrete emotion annotations, and the issues of modal structural differences and cross-sample alignment in multimodal fusion methods persist. To address these, this study undertakes work from both data and model perspectives. For data, a Chinese multimodal database STEM-E2VA was constructed, synchronously collecting four modalities of data: articulatory kinematics, acoustics, glottal signals, and videos. This covers seven discrete emotion categories and employs PAD continuous annotation. By integrating discrete and continuous dimensional annotations, it better represents the distinction between strong and weak emotions under the same discrete emotion label. Concurrently, to process the biases in PAD annotations, we employed the SCL-90 psychological questionnaire to analyze annotators’ cognitive and emotional perceptions, thereby ensuring data reliability. For model, this paper proposes a multimodal supervised contrastive fusion network incorporating PAD perception. It employs a PAD-enhanced hybrid contrastive loss function to optimize intra-model and inter-modal feature alignment. Utilizing a cross-attention mechanism combined with a GRU–Transformer network for temporal feature extraction, it achieves deep fusion of multimodal information, reducing inter-modal discrepancies and cross-class confusion. Experiments demonstrate that the proposed method achieves 85.47% accuracy in discrete sentiment recognition on STEM-E2VA, with a substantial reduction in RMSE for PAD dimension prediction. It also exhibits excellent generalization capability on IEMOCAP, providing a novel framework for integrating discrete and continuous sentiment representations.
- 2 April 2026
- Published by MDPI AG in Multimodal Technologies and Interaction
- Vol. 10 (4) , 38
Open AccessPAD-Guided Multimodal Hybrid Contrastive Emotion Recognition upon STEM-E2VA Dataset
There are still challenges in speech emotion recognition, as the representation capability of single-modal information is limited, there are difficulties in capturing continuous emotional transitions in discrete emotion annotations, and the issues of modal structural differences and cross-sample alignment in multimodal fusion methods persist. To address these, this study undertakes work from both data and model perspectives. For data, a Chinese multimodal database STEM-E2VA was constructed, synchronously collecting four modalities of data: articulatory kinematics, acoustics, glottal signals, and videos. This covers seven discrete emotion categories and employs PAD continuous annotation. By integrating discrete and continuous dimensional annotations, it better represents the distinction between strong and weak emotions under the same discrete emotion label. Concurrently, to process the biases in PAD annotations, we employed the SCL-90 psychological questionnaire to analyze annotators’ cognitive and emotional perceptions, thereby ensuring data reliability. For model, this paper proposes a multimodal supervised contrastive fusion network incorporating PAD perception. It employs a PAD-enhanced hybrid contrastive loss function to optimize intra-model and inter-modal feature alignment. Utilizing a cross-attention mechanism combined with a GRU–Transformer network for temporal feature extraction, it achieves deep fusion of multimodal information, reducing inter-modal discrepancies and cross-class confusion. Experiments demonstrate that the proposed method achieves 85.47% accuracy in discrete sentiment recognition on STEM-E2VA, with a substantial reduction in RMSE for PAD dimension prediction. It also exhibits excellent generalization capability on IEMOCAP, providing a novel framework for integrating discrete and continuous sentiment representations.