Jingran Xie
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Jingran Xie has a total of 9 co-authors. Here are the top co-authors by number of shared publications:
- Yunrui Cai
Shenzhen International Graduate School, Tsinghua University, Shenzhen, China
3 shared publications - Zhiyong Wu
Tsinghua Shenzhen International Graduate School, Shenzhen, Guangdong, China
3 shared publications - Runchuan Ye
Shenzhen International Graduate School, Tsinghua University, Shenzhen, China
2 shared publications - Yaoxun Xu
Shenzhen International Graduate School, Tsinghua University, Shenzhen, China
2 shared publications - Yixuan Zhou
Shenzhen International Graduate School, Tsinghua University, Shenzhen, China
2 shared publications - Boshi Tang
Tsinghua University, Shenzhen, China
1 shared publication
Latest Publications
- Open Access
- 28 October 2024
- Published by Association for Computing Machinery (ACM)
- p. 93-97
Robust Representation Learning for Multimodal Emotion Recognition with Contrastive Learning and Mixup
Multimodal emotion recognition (MER) plays a crucial role in user sentiment analysis and enhancing human-computer interaction experiences. However, in real-world scenarios, noise interference such as environmental noise and image blurring is widespread, impacting the model's ability to recognize emotions. Therefore, improving the robustness of MER models to handle complex and variable noise interference is a significant challenge. To address this challenge, we propose a robust representation learning method based on contrastive learning and Mixup data augmentation strategies to stabilize model performance on noisy data. Specifically, we first perform Mixup data augmentation in the representation space of each modality to broaden class decision boundaries and enhance model generalization. Meanwhile, through a contrastive learning strategy, we align the representations of clean and noisy data to improve model robustness. We evaluate our model's performance on the test set of the MER2024 MER-NOISE track. Our model achieves a weighted average F-score score of 82.71%, which is 3.09% higher than the MER2024 Baseline model. This result demonstrates that our method effectively enhances model robustness and generalization.
- 28 October 2024
- Published by Association for Computing Machinery (ACM)
- p. 93-97
Open AccessRobust Representation Learning for Multimodal Emotion Recognition with Contrastive Learning and Mixup
Multimodal emotion recognition (MER) plays a crucial role in user sentiment analysis and enhancing human-computer interaction experiences. However, in real-world scenarios, noise interference such as environmental noise and image blurring is widespread, impacting the model's ability to recognize emotions. Therefore, improving the robustness of MER models to handle complex and variable noise interference is a significant challenge. To address this challenge, we propose a robust representation learning method based on contrastive learning and Mixup data augmentation strategies to stabilize model performance on noisy data. Specifically, we first perform Mixup data augmentation in the representation space of each modality to broaden class decision boundaries and enhance model generalization. Meanwhile, through a contrastive learning strategy, we align the representations of clean and noisy data to improve model robustness. We evaluate our model's performance on the test set of the MER2024 MER-NOISE track. Our model achieves a weighted average F-score score of 82.71%, which is 3.09% higher than the MER2024 Baseline model. This result demonstrates that our method effectively enhances model robustness and generalization.
- Open Access
- 28 October 2024
- Published by Association for Computing Machinery (ACM)
- p. 104-109
Multimodal Emotion Captioning Using Large Language Model with Prompt Engineering
This paper addresses the challenges in MER 2024 by focusing on the Open Vocabulary (OV) task, which extends beyond traditional fixed label space for multimodal emotion recognition. The study emphasizes the use of Large Language Models (LLMs) to interpret and extract emotional information from multimodal inputs, complemented by speech transcription, speech emotion description, and video clues. The paper explores the integration of these features into a prompt fed into a pre-trained LLaMA3-8B model, utilizing prompt engineering to achieve satisfactory results without fine-tuning. This approach bridges the gap between speech, video and text data, leveraging the full potential of LLMs for open-ended emotion recognition tasks and introducing a solution to the field.
- 28 October 2024
- Published by Association for Computing Machinery (ACM)
- p. 104-109
Open AccessMultimodal Emotion Captioning Using Large Language Model with Prompt Engineering
This paper addresses the challenges in MER 2024 by focusing on the Open Vocabulary (OV) task, which extends beyond traditional fixed label space for multimodal emotion recognition. The study emphasizes the use of Large Language Models (LLMs) to interpret and extract emotional information from multimodal inputs, complemented by speech transcription, speech emotion description, and video clues. The paper explores the integration of these features into a prompt fed into a pre-trained LLaMA3-8B model, utilizing prompt engineering to achieve satisfactory results without fine-tuning. This approach bridges the gap between speech, video and text data, leveraging the full potential of LLMs for open-ended emotion recognition tasks and introducing a solution to the field.
- Open Access
- 29 October 2023
- Published by Association for Computing Machinery (ACM)
- p. 13-20
First-order Multi-label Learning with Cross-modal Interactions for Multimodal Emotion Recognition
Multimodal emotion recognition (MER) is essential for the machine to fully understand human intentions. Various deep neural network based models are proposed but it is still challenging to better model and fuse multimodal features. In addition, recent studies have focused on the classification task of predicting discrete labels, while lacking consideration of the dimension value. In this paper, we propose a multimodal fusion model based on Transformer architecture and cross-modal interactions, and adopt a multi-label learning algorithm of first-order strategy to predict discrete labels and dimension values respectively. We also propose a semi-supervised learning method of moment injection with unlabeled data to enhance the robustness of the model. Finally, we use ensemble learning to further improve the performance of the model. We evaluate the proposed method on the MER-MULTI sub-challenge of Multimodal Emotion Recognition Challenge (MER 2023). Experimental results demonstrate the promising performance of our proposed method, which can achieve the evaluation metric of 0.6765 on the test set.
- 29 October 2023
- Published by Association for Computing Machinery (ACM)
- p. 13-20
Open AccessFirst-order Multi-label Learning with Cross-modal Interactions for Multimodal Emotion Recognition
Multimodal emotion recognition (MER) is essential for the machine to fully understand human intentions. Various deep neural network based models are proposed but it is still challenging to better model and fuse multimodal features. In addition, recent studies have focused on the classification task of predicting discrete labels, while lacking consideration of the dimension value. In this paper, we propose a multimodal fusion model based on Transformer architecture and cross-modal interactions, and adopt a multi-label learning algorithm of first-order strategy to predict discrete labels and dimension values respectively. We also propose a semi-supervised learning method of moment injection with unlabeled data to enhance the robustness of the model. Finally, we use ensemble learning to further improve the performance of the model. We evaluate the proposed method on the MER-MULTI sub-challenge of Multimodal Emotion Recognition Challenge (MER 2023). Experimental results demonstrate the promising performance of our proposed method, which can achieve the evaluation metric of 0.6765 on the test set.