Guimin Hu
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Guimin Hu has a total of 19 co-authors. Here are the top co-authors by number of shared publications:
- Hasti Seifi
Arizona State University, Tempe, Arizona, USA
5 shared publications - Guangming Lu
Harbin Institute of Technology, Shenzhen, Shenzhen, China
4 shared publications - Yi Zhao
School of Science, Harbin Institute of Technology, Shenzhen, China
4 shared publications - Daniel Hershcovich
University of Copenhagen
3 shared publications Lijie Hu
Mohamed Zayed University of Artificial Intelligence
2 shared publicationsZhihong Zhu
Tencent
2 shared publications
Latest Publications
- Open Access
- 4 June 2026
- Published by Elsevier in SSRN Electronic Journal
Label-oriented Hierarchical Framework for Multimodal Multi-label Emotion Recognition
Multimodal multi-label emotion recognition (MMER) aims to identify one or more emotion labels using multimodal inputs. However, most existing methods mainly model label dependencies while overlooking the hierarchical structure of MMER and the label consistency across modalities. Since different modalities provide emotional cues with varying strength, our framework leverages this asymmetry: hierarchical voting adaptively aggregates complementary cues from modalities and modality segments, reducing ambiguity. We conceptualize MMER as a hierarchical prediction process and propose a hierarchical voting framework that collects and aggregates decisions from both whole-modality representations and modality segments. We further introduce modality swap prediction to enhance label consistency and cross-modal complementarity. Experiments on English benchmark datasets CMU-MOSEI and MER2023, as well as the Chinese benchmark M3ED achieve state-of-the-art results, demonstrating the competitiveness and effectiveness of our approach.
- 4 June 2026
- Published by Elsevier in SSRN Electronic Journal
Open AccessLabel-oriented Hierarchical Framework for Multimodal Multi-label Emotion Recognition
Multimodal multi-label emotion recognition (MMER) aims to identify one or more emotion labels using multimodal inputs. However, most existing methods mainly model label dependencies while overlooking the hierarchical structure of MMER and the label consistency across modalities. Since different modalities provide emotional cues with varying strength, our framework leverages this asymmetry: hierarchical voting adaptively aggregates complementary cues from modalities and modality segments, reducing ambiguity. We conceptualize MMER as a hierarchical prediction process and propose a hierarchical voting framework that collects and aggregates decisions from both whole-modality representations and modality segments. We further introduce modality swap prediction to enhance label consistency and cross-modal complementarity. Experiments on English benchmark datasets CMU-MOSEI and MER2023, as well as the Chinese benchmark M3ED achieve state-of-the-art results, demonstrating the competitiveness and effectiveness of our approach.
- Open Access
- 27 October 2025
- Published by Association for Computing Machinery (ACM)
- p. 2362-2371
PgM: Partitioner Guided Modal Learning Framework
- 27 October 2025
- Published by Association for Computing Machinery (ACM)
- p. 2362-2371
Open AccessPgM: Partitioner Guided Modal Learning Framework
- Open Access
HapticCap: A Multimodal Dataset and Task for Understanding User Experience of Vibration Haptic Signals
Haptic signals, from smartphone vibrations to virtual reality touch feedback, can effectively convey information and enhance realism, but designing signals that resonate meaningfully with users is challenging. To facilitate this, we introduce a multimodal dataset and task, of matching user descriptions to vibration haptic signals, and highlight two primary challenges: (1) lack of large haptic vibration datasets annotated with textual descriptions as collecting haptic descriptions is time-consuming, and (2) limited capability of existing tasks and models to describe vibration signals in text. To advance this area, we create HapticCap, the first fully human-annotated haptic-captioned dataset, containing 92,070 haptic-text pairs for user descriptions of sensory, emotional, and associative attributes of vibrations. Based on HapticCap, we propose the haptic-caption retrieval task and present the results of this task from a supervised contrastive learning framework that brings together text representations within specific categories and vibrations. Overall, the combination of language model T5 and audio model AST yields the best performance in the haptic-caption retrieval task, especially when separately trained for each description category.
Open AccessHapticCap: A Multimodal Dataset and Task for Understanding User Experience of Vibration Haptic Signals
Haptic signals, from smartphone vibrations to virtual reality touch feedback, can effectively convey information and enhance realism, but designing signals that resonate meaningfully with users is challenging. To facilitate this, we introduce a multimodal dataset and task, of matching user descriptions to vibration haptic signals, and highlight two primary challenges: (1) lack of large haptic vibration datasets annotated with textual descriptions as collecting haptic descriptions is time-consuming, and (2) limited capability of existing tasks and models to describe vibration signals in text. To advance this area, we create HapticCap, the first fully human-annotated haptic-captioned dataset, containing 92,070 haptic-text pairs for user descriptions of sensory, emotional, and associative attributes of vibrations. Based on HapticCap, we propose the haptic-caption retrieval task and present the results of this task from a supervised contrastive learning framework that brings together text representations within specific categories and vibrations. Overall, the combination of language model T5 and audio model AST yields the best performance in the haptic-caption retrieval task, especially when separately trained for each description category.
- Open Access
Partitioner Guided Modal Learning Framework
Multimodal learning benefits from multiple modal information, and each learned modal representations can be divided into uni-modal that can be learned from uni-modal training and paired-modal features that can be learned from cross-modal interaction. Building on this perspective, we propose a partitioner-guided modal learning framework, PgM, which consists of the modal partitioner, uni-modal learner, paired-modal learner, and uni-paired modal decoder. Modal partitioner segments the learned modal representation into uni-modal and paired-modal features. Modal learner incorporates two dedicated components for uni-modal and paired-modal learning. Uni-paired modal decoder reconstructs modal representation based on uni-modal and paired-modal features. PgM offers three key benefits: 1) thorough learning of uni-modal and paired-modal features, 2) flexible distribution adjustment for uni-modal and paired-modal representations to suit diverse downstream tasks, and 3) different learning rates across modalities and partitions. Extensive experiments demonstrate the effectiveness of PgM across four multimodal tasks and further highlight its transferability to existing models. Additionally, we visualize the distribution of uni-modal and paired-modal features across modalities and tasks, offering insights into their respective contributions.
Open AccessPartitioner Guided Modal Learning Framework
Multimodal learning benefits from multiple modal information, and each learned modal representations can be divided into uni-modal that can be learned from uni-modal training and paired-modal features that can be learned from cross-modal interaction. Building on this perspective, we propose a partitioner-guided modal learning framework, PgM, which consists of the modal partitioner, uni-modal learner, paired-modal learner, and uni-paired modal decoder. Modal partitioner segments the learned modal representation into uni-modal and paired-modal features. Modal learner incorporates two dedicated components for uni-modal and paired-modal learning. Uni-paired modal decoder reconstructs modal representation based on uni-modal and paired-modal features. PgM offers three key benefits: 1) thorough learning of uni-modal and paired-modal features, 2) flexible distribution adjustment for uni-modal and paired-modal representations to suit diverse downstream tasks, and 3) different learning rates across modalities and partitions. Extensive experiments demonstrate the effectiveness of PgM across four multimodal tasks and further highlight its transferability to existing models. Additionally, we visualize the distribution of uni-modal and paired-modal features across modalities and tasks, offering insights into their respective contributions.
- 7 January 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Audio, Speech and Language Processing
- Vol. 33, 1974-1986
Improving Representation With Intra- and Inter-Clause Commonsense Injection for Emotion-Cause Pair Extraction
Emotion-Cause Pair Extraction (ECPE) aims to simultaneously extract emotions and their corresponding causes, which is crucial for understanding human intent. Generally, machines can extract or infer emotion causes by incorporating commonsense knowledge with document contexts. However, most existing works lack integration of commonsense knowledge when analyzing human emotions and their triggering causes. To address this issue, this paper introduces a commonsense knowledge-injection framework (CokECPE), designed to enhance representation learning and thereby improve model performance. CokECPE enriches representation learning by incorporating commonsense knowledge at intra-clause and inter-clause levels. First, CokECPE retrieves concept-level and event-level knowledge from the commonsense conversational knowledge graph $\text{C}^{3}\text{KG}$ . Next, it injects these commonsense features into a pretrained model, fusing them with multilevel textual information. Finally, CokECPE establishes dependencies between clauses based on commonsense reasoning, selecting the paths most similar to the emotion-cause pair candidates as edges, effectively embedding commonsense relations into the representation learning process. Experimental results on two public datasets demonstrate that CokECPE consistently outperforms existing methods, achieving enhanced state-of-the-art performance and validating its effectiveness.
- 7 January 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Audio, Speech and Language Processing
- Vol. 33, 1974-1986
Improving Representation With Intra- and Inter-Clause Commonsense Injection for Emotion-Cause Pair Extraction
Emotion-Cause Pair Extraction (ECPE) aims to simultaneously extract emotions and their corresponding causes, which is crucial for understanding human intent. Generally, machines can extract or infer emotion causes by incorporating commonsense knowledge with document contexts. However, most existing works lack integration of commonsense knowledge when analyzing human emotions and their triggering causes. To address this issue, this paper introduces a commonsense knowledge-injection framework (CokECPE), designed to enhance representation learning and thereby improve model performance. CokECPE enriches representation learning by incorporating commonsense knowledge at intra-clause and inter-clause levels. First, CokECPE retrieves concept-level and event-level knowledge from the commonsense conversational knowledge graph $\text{C}^{3}\text{KG}$ . Next, it injects these commonsense features into a pretrained model, fusing them with multilevel textual information. Finally, CokECPE establishes dependencies between clauses based on commonsense reasoning, selecting the paths most similar to the emotion-cause pair candidates as edges, effectively embedding commonsense relations into the representation learning process. Experimental results on two public datasets demonstrate that CokECPE consistently outperforms existing methods, achieving enhanced state-of-the-art performance and validating its effectiveness.
- Open Access
- 1 January 2025
- Published by Association for Computational Linguistics (ACL)
- p. 14473-14489
HapticCap: A Multimodal Dataset and Task for Understanding User Experience of Vibration Haptic Signals
Guimin Hu, Daniel Hershcovich, Hasti Seifi. Findings of the Association for Computational Linguistics: EMNLP 2025. 2025.
- 1 January 2025
- Published by Association for Computational Linguistics (ACL)
- p. 14473-14489
Open AccessHapticCap: A Multimodal Dataset and Task for Understanding User Experience of Vibration Haptic Signals
Guimin Hu, Daniel Hershcovich, Hasti Seifi. Findings of the Association for Computational Linguistics: EMNLP 2025. 2025.
- Open Access
Grounding Emotional Descriptions to Electrovibration Haptic Signals
Designing and displaying haptic signals with sensory and emotional attributes can improve the user experience in various applications. Free-form user language provides rich sensory and emotional information for haptic design (e.g., ``This signal feels smooth and exciting''), but little work exists on linking user descriptions to haptic signals (i.e., language grounding). To address this gap, we conducted a study where 12 users described the feel of 32 signals perceived on a surface haptics (i.e., electrovibration) display. We developed a computational pipeline using natural language processing (NLP) techniques, such as GPT-3.5 Turbo and word embedding methods, to extract sensory and emotional keywords and group them into semantic clusters (i.e., concepts). We linked the keyword clusters to haptic signal features (e.g., pulse count) using correlation analysis. The proposed pipeline demonstrates the viability of a computational approach to analyzing haptic experiences. We discuss our future plans for creating a predictive model of haptic experience.
Open AccessGrounding Emotional Descriptions to Electrovibration Haptic Signals
Designing and displaying haptic signals with sensory and emotional attributes can improve the user experience in various applications. Free-form user language provides rich sensory and emotional information for haptic design (e.g., ``This signal feels smooth and exciting''), but little work exists on linking user descriptions to haptic signals (i.e., language grounding). To address this gap, we conducted a study where 12 users described the feel of 32 signals perceived on a surface haptics (i.e., electrovibration) display. We developed a computational pipeline using natural language processing (NLP) techniques, such as GPT-3.5 Turbo and word embedding methods, to extract sensory and emotional keywords and group them into semantic clusters (i.e., concepts). We linked the keyword clusters to haptic signal features (e.g., pulse count) using correlation analysis. The proposed pipeline demonstrates the viability of a computational approach to analyzing haptic experiences. We discuss our future plans for creating a predictive model of haptic experience.
- 5 June 2024
- Published by Elsevier in Neural Networks
- Vol. 178, 106431
Unifying emotion-oriented and cause-oriented predictions for emotion-cause pair extraction
- 5 June 2024
- Published by Elsevier in Neural Networks
- Vol. 178, 106431
Unifying emotion-oriented and cause-oriented predictions for emotion-cause pair extraction
- 29 April 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Affective Computing
- Vol. 15 (4) , 1997-2011
Improving Representation With Hierarchical Contrastive Learning for Emotion-Cause Pair Extraction
Emotion-cause pair extraction (ECPE) aims to extract emotions and their corresponding cause from a document. The previous works have made great progress. However, there exist two major issues in existing works. First, most existing works mainly focus on the semantic relation between the emotion clause and cause clause, ignoring their inner statistical relation in representation space. Second, the existing works are sensitive to the relative position between the emotion clause and cause clause, which damages the model's robustness. To address the two issues, we propose a hierarchical contrastive learning framework (HCL-ECPE), which hierarchically performs contrastive learning on representation from two levels. The first level is inter-clause contrastive learning (ICCL), which performs between emotion clause and cause clause through mutual information maximization. The second level is intra-pair contrastive learning (IPCL), which performs between clause representation and pair representation through contrastive predictive coding (CPC). HCL-ECPE integrates ICCL and IPCL modules to explore the statistical relations between the emotion clause, cause clause, and their constructed emotion-cause pair from the perspective of mutual information, thereby improving the model performance and robustness. Experimental results on two public datasets, ECPED and RECCON, demonstrate that HCL-ECPE outperforms the most competitive baselines. Furthermore, ICCL and IPCL are orthogonal to the existing model, and introducing them into the current models updates state-of-the-art performance.
- 29 April 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Affective Computing
- Vol. 15 (4) , 1997-2011
Improving Representation With Hierarchical Contrastive Learning for Emotion-Cause Pair Extraction
Emotion-cause pair extraction (ECPE) aims to extract emotions and their corresponding cause from a document. The previous works have made great progress. However, there exist two major issues in existing works. First, most existing works mainly focus on the semantic relation between the emotion clause and cause clause, ignoring their inner statistical relation in representation space. Second, the existing works are sensitive to the relative position between the emotion clause and cause clause, which damages the model's robustness. To address the two issues, we propose a hierarchical contrastive learning framework (HCL-ECPE), which hierarchically performs contrastive learning on representation from two levels. The first level is inter-clause contrastive learning (ICCL), which performs between emotion clause and cause clause through mutual information maximization. The second level is intra-pair contrastive learning (IPCL), which performs between clause representation and pair representation through contrastive predictive coding (CPC). HCL-ECPE integrates ICCL and IPCL modules to explore the statistical relations between the emotion clause, cause clause, and their constructed emotion-cause pair from the perspective of mutual information, thereby improving the model performance and robustness. Experimental results on two public datasets, ECPED and RECCON, demonstrate that HCL-ECPE outperforms the most competitive baselines. Furthermore, ICCL and IPCL are orthogonal to the existing model, and introducing them into the current models updates state-of-the-art performance.
- Open Access
- 1 January 2024
- Published by Association for Computational Linguistics (ACL)
- p. 5248-5261
UniMEEC: Towards Unified Multimodal Emotion Recognition and Emotion Cause
Guimin Hu, Zhihong Zhu, Daniel Hershcovich, Lijie Hu, Hasti Seifi, Jiayuan Xie. Findings of the Association for Computational Linguistics: EMNLP 2024. 2024.
- 1 January 2024
- Published by Association for Computational Linguistics (ACL)
- p. 5248-5261
Open AccessUniMEEC: Towards Unified Multimodal Emotion Recognition and Emotion Cause
Guimin Hu, Zhihong Zhu, Daniel Hershcovich, Lijie Hu, Hasti Seifi, Jiayuan Xie. Findings of the Association for Computational Linguistics: EMNLP 2024. 2024.