GHGuimin Hu

Guimin Hu

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article

Previous Organizations / Institutes

The organizations / institutes from the scholar's other articles in the past ten years


Guimin Hu has a total of 19 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Label-oriented Hierarchical Framework for Multimodal Multi-label Emotion Recognition

    by Guimin Hu, Meng Li, Yi Zhao, Guangming Lu

    Multimodal multi-label emotion recognition (MMER) aims to identify one or more emotion labels using multimodal inputs. However, most existing methods mainly model label dependencies while overlooking the hierarchical structure of MMER and the label consistency across modalities. Since different modalities provide emotional cues with varying strength, our framework leverages this asymmetry: hierarchical voting adaptively aggregates complementary cues from modalities and modality segments, reducing ambiguity. We conceptualize MMER as a hierarchical prediction process and propose a hierarchical voting framework that collects and aggregates decisions from both whole-modality representations and modality segments. We further introduce modality swap prediction to enhance label consistency and cross-modal complementarity. Experiments on English benchmark datasets CMU-MOSEI and MER2023, as well as the Chinese benchmark M3ED achieve state-of-the-art results, demonstrating the competitiveness and effectiveness of our approach.

  • Open Access

    PgM: Partitioner Guided Modal Learning Framework

    by Guimin Hu, Yi Xin, Lijie Hu, Zhihong Zhu, Hasti Seifi


  • Open Access

    HapticCap: A Multimodal Dataset and Task for Understanding User Experience of Vibration Haptic Signals

    by Guimin Hu, Daniel Hershcovich, Hasti Seifi

    Haptic signals, from smartphone vibrations to virtual reality touch feedback, can effectively convey information and enhance realism, but designing signals that resonate meaningfully with users is challenging. To facilitate this, we introduce a multimodal dataset and task, of matching user descriptions to vibration haptic signals, and highlight two primary challenges: (1) lack of large haptic vibration datasets annotated with textual descriptions as collecting haptic descriptions is time-consuming, and (2) limited capability of existing tasks and models to describe vibration signals in text. To advance this area, we create HapticCap, the first fully human-annotated haptic-captioned dataset, containing 92,070 haptic-text pairs for user descriptions of sensory, emotional, and associative attributes of vibrations. Based on HapticCap, we propose the haptic-caption retrieval task and present the results of this task from a supervised contrastive learning framework that brings together text representations within specific categories and vibrations. Overall, the combination of language model T5 and audio model AST yields the best performance in the haptic-caption retrieval task, especially when separately trained for each description category.


  • Open Access

    Partitioner Guided Modal Learning Framework

    by Guimin Hu, Yi Xin, Lijie Hu, Zhihong Zhu, Hasti Seifi

    Multimodal learning benefits from multiple modal information, and each learned modal representations can be divided into uni-modal that can be learned from uni-modal training and paired-modal features that can be learned from cross-modal interaction. Building on this perspective, we propose a partitioner-guided modal learning framework, PgM, which consists of the modal partitioner, uni-modal learner, paired-modal learner, and uni-paired modal decoder. Modal partitioner segments the learned modal representation into uni-modal and paired-modal features. Modal learner incorporates two dedicated components for uni-modal and paired-modal learning. Uni-paired modal decoder reconstructs modal representation based on uni-modal and paired-modal features. PgM offers three key benefits: 1) thorough learning of uni-modal and paired-modal features, 2) flexible distribution adjustment for uni-modal and paired-modal representations to suit diverse downstream tasks, and 3) different learning rates across modalities and partitions. Extensive experiments demonstrate the effectiveness of PgM across four multimodal tasks and further highlight its transferability to existing models. Additionally, we visualize the distribution of uni-modal and paired-modal features across modalities and tasks, offering insights into their respective contributions.

  • Improving Representation With Intra- and Inter-Clause Commonsense Injection for Emotion-Cause Pair Extraction

    by Guimin Hu, Jiayuan Xie

    Emotion-Cause Pair Extraction (ECPE) aims to simultaneously extract emotions and their corresponding causes, which is crucial for understanding human intent. Generally, machines can extract or infer emotion causes by incorporating commonsense knowledge with document contexts. However, most existing works lack integration of commonsense knowledge when analyzing human emotions and their triggering causes. To address this issue, this paper introduces a commonsense knowledge-injection framework (CokECPE), designed to enhance representation learning and thereby improve model performance. CokECPE enriches representation learning by incorporating commonsense knowledge at intra-clause and inter-clause levels. First, CokECPE retrieves concept-level and event-level knowledge from the commonsense conversational knowledge graph $\text{C}^{3}\text{KG}$ . Next, it injects these commonsense features into a pretrained model, fusing them with multilevel textual information. Finally, CokECPE establishes dependencies between clauses based on commonsense reasoning, selecting the paths most similar to the emotion-cause pair candidates as edges, effectively embedding commonsense relations into the representation learning process. Experimental results on two public datasets demonstrate that CokECPE consistently outperforms existing methods, achieving enhanced state-of-the-art performance and validating its effectiveness.


  • Open Access

    HapticCap: A Multimodal Dataset and Task for Understanding User Experience of Vibration Haptic Signals

    by Guimin Hu, Daniel Hershcovich, Hasti Seifi

    Guimin Hu, Daniel Hershcovich, Hasti Seifi. Findings of the Association for Computational Linguistics: EMNLP 2025. 2025.


  • Open Access

    Grounding Emotional Descriptions to Electrovibration Haptic Signals

    by Guimin Hu, Zirui Zhao, Lukas Heilmann, Yasemin Vardar, Hasti Seifi

    Designing and displaying haptic signals with sensory and emotional attributes can improve the user experience in various applications. Free-form user language provides rich sensory and emotional information for haptic design (e.g., ``This signal feels smooth and exciting''), but little work exists on linking user descriptions to haptic signals (i.e., language grounding). To address this gap, we conducted a study where 12 users described the feel of 32 signals perceived on a surface haptics (i.e., electrovibration) display. We developed a computational pipeline using natural language processing (NLP) techniques, such as GPT-3.5 Turbo and word embedding methods, to extract sensory and emotional keywords and group them into semantic clusters (i.e., concepts). We linked the keyword clusters to haptic signal features (e.g., pulse count) using correlation analysis. The proposed pipeline demonstrates the viability of a computational approach to analyzing haptic experiences. We discuss our future plans for creating a predictive model of haptic experience.

  • Improving Representation With Hierarchical Contrastive Learning for Emotion-Cause Pair Extraction

    by Guimin Hu, Yi Zhao, Guangming Lu

    Emotion-cause pair extraction (ECPE) aims to extract emotions and their corresponding cause from a document. The previous works have made great progress. However, there exist two major issues in existing works. First, most existing works mainly focus on the semantic relation between the emotion clause and cause clause, ignoring their inner statistical relation in representation space. Second, the existing works are sensitive to the relative position between the emotion clause and cause clause, which damages the model's robustness. To address the two issues, we propose a hierarchical contrastive learning framework (HCL-ECPE), which hierarchically performs contrastive learning on representation from two levels. The first level is inter-clause contrastive learning (ICCL), which performs between emotion clause and cause clause through mutual information maximization. The second level is intra-pair contrastive learning (IPCL), which performs between clause representation and pair representation through contrastive predictive coding (CPC). HCL-ECPE integrates ICCL and IPCL modules to explore the statistical relations between the emotion clause, cause clause, and their constructed emotion-cause pair from the perspective of mutual information, thereby improving the model performance and robustness. Experimental results on two public datasets, ECPED and RECCON, demonstrate that HCL-ECPE outperforms the most competitive baselines. Furthermore, ICCL and IPCL are orthogonal to the existing model, and introducing them into the current models updates state-of-the-art performance.


  • Open Access

    UniMEEC: Towards Unified Multimodal Emotion Recognition and Emotion Cause

    by Guimin Hu, Zhihong Zhu, Daniel Hershcovich, Lijie Hu, Hasti Seifi, Jiayuan Xie

    Guimin Hu, Zhihong Zhu, Daniel Hershcovich, Lijie Hu, Hasti Seifi, Jiayuan Xie. Findings of the Association for Computational Linguistics: EMNLP 2024. 2024.