Yingjian Li
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Yingjian Li has a total of 46 co-authors. Here are the top co-authors by number of shared publications:
- Guangming Lu
Harbin Institute of Technology, Shenzhen, Shenzhen, China
13 shared publications - Zheng Zhang
University of California Santa Barbara
8 shared publications - Jinxing Li
Harbin Institute of Technology
7 shared publications - David Zhang
The Chinese University of Hong Kong
5 shared publications - Yaowei Wang
Harbin Institute of Technology (Shenzhen), Shenzhen, China and Pengcheng Laboratory, Shenzhen, China
5 shared publications - Bingzhi Chen
Beijing Institute of Technology, Zhuhai, Zhuhai, China
4 shared publications
Latest Publications
- 1 November 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 48  (11) , 15412-15426
PolaFormer++: Polarity-Aware Linear Attention With Channel-Wise Spikiness
- 1 November 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 48  (11) , 15412-15426
PolaFormer++: Polarity-Aware Linear Attention With Channel-Wise Spikiness
- 1 June 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. PP  (15209210) , 1-15
CRNet: Context-Guided Recovering Network for Incomplete Multimodal Emotion Recognition in Conversation
Incomplete multimodal learning has become a critical challenge in Emotion Recognition in Conversation (ERC). However, most methods focus on individual utterances instead of conversational data, thus often overlooking the rich contextual information embedded within conversations. Moreover, existing incomplete ERC approaches are typically limited to joint multimodal representation learning for missing modality recovery, failing to account for the complementary information inherent in modality-specific characteristics. In this paper, we propose a Context-guided Recovering Network (CRNet) to comprehensively explore both modality characteristics and contextual information in conversational data for incomplete multimodal ERC. Specifically, to effectively leverage cross-modal semantic consistency while preserving the modality-specific information, we design a context consistent recovery module. This module uses the semantic relations between utterances to guide the recovery of missing components. Furthermore, to maintain the balance between preserving emotion consistency and uncovering emotion shifts in contextual modeling, we introduce a dual-level semantic learning module, which is composed of the instance-level contrastive unit and label-based emotion shift unit. The instance-level contrastive unit enforces similarity constraints across recovered instances within each modality, while the label-based emotion shift unit captures emotional variations in multimodal fusion representations. Experimental results on four benchmark datasets demonstrate that the proposed CRNet achieves remarkable performance compared with state-of-the-art baselines.
- 1 June 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. PP  (15209210) , 1-15
CRNet: Context-Guided Recovering Network for Incomplete Multimodal Emotion Recognition in Conversation
Incomplete multimodal learning has become a critical challenge in Emotion Recognition in Conversation (ERC). However, most methods focus on individual utterances instead of conversational data, thus often overlooking the rich contextual information embedded within conversations. Moreover, existing incomplete ERC approaches are typically limited to joint multimodal representation learning for missing modality recovery, failing to account for the complementary information inherent in modality-specific characteristics. In this paper, we propose a Context-guided Recovering Network (CRNet) to comprehensively explore both modality characteristics and contextual information in conversational data for incomplete multimodal ERC. Specifically, to effectively leverage cross-modal semantic consistency while preserving the modality-specific information, we design a context consistent recovery module. This module uses the semantic relations between utterances to guide the recovery of missing components. Furthermore, to maintain the balance between preserving emotion consistency and uncovering emotion shifts in contextual modeling, we introduce a dual-level semantic learning module, which is composed of the instance-level contrastive unit and label-based emotion shift unit. The instance-level contrastive unit enforces similarity constraints across recovered instances within each modality, while the label-based emotion shift unit captures emotional variations in multimodal fusion representations. Experimental results on four benchmark datasets demonstrate that the proposed CRNet achieves remarkable performance compared with state-of-the-art baselines.
- 9 March 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 36  (7) , 9888-9900
Rethinking the Knowledge Gap Between Cloud and Device Models for Effective Co-Adaptation
By collaboratively updating the cloud (large-scale) and device (small-scale) models, co-adaptation aims to enhance the generalization performance of device models in response to the distribution shifts in the incoming data. Existing methods often rely on low-entropy samples that are selected by the device model for co-adaptation, which ignores the differences between the predictions of the cloud and device models that are caused by the knowledge gap. As a result, some of the selected samples are redundant and contribute limited value to cloud model updating and knowledge distillation. To this end, we propose a test-time co-adaptation method by Rethinking the Knowledge Gap (RKG) between cloud and device models, which effectively updates the models by informative sample selection and targeted knowledge distillation for image-based classification tasks. Specifically, we design a sample selection module that integrates semantic prediction entropy with object structure cues to identify valuable samples, which effectively alleviates the redundancy problem. Based on these selected samples, we further construct a reweighting module that measures the prediction consistency between the two models and assigns greater emphasis to samples with larger prediction discrepancies, i.e., larger knowledge gaps, to improve knowledge distillation. Furthermore, by jointly leveraging these two modules, RKG enables efficient and effective co-adaptation, thereby achieving robust model generalization to continuously changing data in classification scenarios. Extensive experiments demonstrate that RKG outperforms state-of-the-art methods while requiring fewer uploaded samples.
- 9 March 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 36  (7) , 9888-9900
Rethinking the Knowledge Gap Between Cloud and Device Models for Effective Co-Adaptation
By collaboratively updating the cloud (large-scale) and device (small-scale) models, co-adaptation aims to enhance the generalization performance of device models in response to the distribution shifts in the incoming data. Existing methods often rely on low-entropy samples that are selected by the device model for co-adaptation, which ignores the differences between the predictions of the cloud and device models that are caused by the knowledge gap. As a result, some of the selected samples are redundant and contribute limited value to cloud model updating and knowledge distillation. To this end, we propose a test-time co-adaptation method by Rethinking the Knowledge Gap (RKG) between cloud and device models, which effectively updates the models by informative sample selection and targeted knowledge distillation for image-based classification tasks. Specifically, we design a sample selection module that integrates semantic prediction entropy with object structure cues to identify valuable samples, which effectively alleviates the redundancy problem. Based on these selected samples, we further construct a reweighting module that measures the prediction consistency between the two models and assigns greater emphasis to samples with larger prediction discrepancies, i.e., larger knowledge gaps, to improve knowledge distillation. Furthermore, by jointly leveraging these two modules, RKG enables efficient and effective co-adaptation, thereby achieving robust model generalization to continuously changing data in classification scenarios. Extensive experiments demonstrate that RKG outperforms state-of-the-art methods while requiring fewer uploaded samples.
- 13 February 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks and Learning Systems
- Vol. PP  (2162237X) , 1-15
Class-Incremental Cloud-Device Collaborative Adaptation With Contrastive Learning in Dynamic Changing Environments
Lightweight models are often deployed on edge devices (e.g., smartphones and wearable devices) to enhance their scalability and practicability. To enhance their generalization ability in dynamic changing environments, cloud-device collaborative learning (CDCL) is proposed to transfer the generalization ability from large models on cloud servers to lightweight models deployed on devices. However, current methods mainly focus on solving the data distribution shifts for a limited number of seen classes but ignore the continual incoming new classes. Though existing class-incremental learning (CIL) methods achieve impressive performance, two major challenges arise when adapting them into the CDCL setting: 1) poor generalization of lightweight models during CIL and 2) overfitting during data-incremental learning. In this article, we explore a new problem named class-incremental CI-CDCL, and propose a contrastive prototypical network based CI-CDCL framework, aiming to improve the effectiveness of cloud-device collaboration in both class-incremental and data-incremental learning. Extensive experiments are conducted on two public datasets and the experimental results can evaluate the effectiveness of our proposed model.
- 13 February 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks and Learning Systems
- Vol. PP  (2162237X) , 1-15
Class-Incremental Cloud-Device Collaborative Adaptation With Contrastive Learning in Dynamic Changing Environments
Lightweight models are often deployed on edge devices (e.g., smartphones and wearable devices) to enhance their scalability and practicability. To enhance their generalization ability in dynamic changing environments, cloud-device collaborative learning (CDCL) is proposed to transfer the generalization ability from large models on cloud servers to lightweight models deployed on devices. However, current methods mainly focus on solving the data distribution shifts for a limited number of seen classes but ignore the continual incoming new classes. Though existing class-incremental learning (CIL) methods achieve impressive performance, two major challenges arise when adapting them into the CDCL setting: 1) poor generalization of lightweight models during CIL and 2) overfitting during data-incremental learning. In this article, we explore a new problem named class-incremental CI-CDCL, and propose a contrastive prototypical network based CI-CDCL framework, aiming to improve the effectiveness of cloud-device collaboration in both class-incremental and data-incremental learning. Extensive experiments are conducted on two public datasets and the experimental results can evaluate the effectiveness of our proposed model.
- 9 January 2026
- Published by Elsevier in Information Processing & Management
- Vol. 63  (4)
Modality augmentation and task-aware dual-modal LoRAs for multi-task multimodal federated learning
- 9 January 2026
- Published by Elsevier in Information Processing & Management
- Vol. 63  (4)
Modality augmentation and task-aware dual-modal LoRAs for multi-task multimodal federated learning
- 23 September 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Affective Computing
- Vol. 17  (1) , 290-302
Hierarchical Multi-Criteria Representation Fusion for Robust Incomplete Multimodal Sentiment Analysis
The challenge of improving robustness to missing data in multimodal sentiment analysis (MSA) has recently attracted increasing attention. The existing work is based on sentence-level sentiment analysis, which requires extracting vector representations from input multimodal sentence sequences to perform sentiment regression or classification tasks. However, most studies focus primarily on exploring cross-modal correlations, neglecting the dynamic semantic variations across sequence frames during feature merging and fusion. In this paper, we propose the Hierarchical Multi-criteria Representation Fusion (HMRF) framework, which effectively captures dynamic semantic information across frames and modalities. Specifically, HMRF consists of two parts, i.e., sequence merging and information transfer. For sequence merging, we design a hierarchical cross-modal semantic perception mechanism and a multi-criteria feature merging mechanism, which together mitigate unimodal affective bias and progressively integrate incomplete feature sequences. Moreover, for information transfer, hierarchical information distillation and high-level semantic reconstruction are performed to transfer certain attributes and sentiment information from the complete view to the incomplete view during training. Extensive experiments on multiple benchmark datasets consistently demonstrate that the proposed HMRF outperforms existing baselines.
- 23 September 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Affective Computing
- Vol. 17  (1) , 290-302
Hierarchical Multi-Criteria Representation Fusion for Robust Incomplete Multimodal Sentiment Analysis
The challenge of improving robustness to missing data in multimodal sentiment analysis (MSA) has recently attracted increasing attention. The existing work is based on sentence-level sentiment analysis, which requires extracting vector representations from input multimodal sentence sequences to perform sentiment regression or classification tasks. However, most studies focus primarily on exploring cross-modal correlations, neglecting the dynamic semantic variations across sequence frames during feature merging and fusion. In this paper, we propose the Hierarchical Multi-criteria Representation Fusion (HMRF) framework, which effectively captures dynamic semantic information across frames and modalities. Specifically, HMRF consists of two parts, i.e., sequence merging and information transfer. For sequence merging, we design a hierarchical cross-modal semantic perception mechanism and a multi-criteria feature merging mechanism, which together mitigate unimodal affective bias and progressively integrate incomplete feature sequences. Moreover, for information transfer, hierarchical information distillation and high-level semantic reconstruction are performed to transfer certain attributes and sentiment information from the complete view to the incomplete view during training. Extensive experiments on multiple benchmark datasets consistently demonstrate that the proposed HMRF outperforms existing baselines.
- 21 October 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Instrumentation and Measurement
- Vol. 74  (00189456) , 1-15
Triple-Stream Siamese Segmentation Network for Printed Label Defect Detection
Detecting defects in printed labels is essential for quality control. Although a few of vision-based models have been proposed for this challenging task, they fail to deal with the large deformation in printed labels, and can not be generalized well to unseen defects. To this end, we propose a novel Triple-stream Siamese Segmentation Network (TSS-Net) to overcome these issues. TSS-Net utilizes two separate Siamese groups integrated with a registration module to learn differential features based on Siamese similarity comparison, and employs a feature complementary learning strategy, so that the model is able to simultaneously handle large deformations and generalize to unseen defects. Specifically, a pre-trained Registration Module (RM) is integrated into the triple-stream Siamese segmentation network to achieve end-to-end detection, which enhances the robustness to large deformations. Moreover, a group of Differential Feature Enhancement (DFE) modules are constructed to learn differential features at different scales, which forces the network to focus on changed features while ignores invariant features. Further, a feature complementary learning strategy is designed to fuse differential features at different scales, which can suppress artifacts during network reconstruction. Finally, a method is proposed to simulate printed labels with varying degrees of deformation and artificial defects. Extensive experimental results show that our proposed TSS-Net yields the best performance compared with the state-of-the-art method. Specifically, our proposed method achieves improvements of 4.20% on the F 1 score and 7.51% on the Intersection over Union (IoU).
- 21 October 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Instrumentation and Measurement
- Vol. 74  (00189456) , 1-15
Triple-Stream Siamese Segmentation Network for Printed Label Defect Detection
Detecting defects in printed labels is essential for quality control. Although a few of vision-based models have been proposed for this challenging task, they fail to deal with the large deformation in printed labels, and can not be generalized well to unseen defects. To this end, we propose a novel Triple-stream Siamese Segmentation Network (TSS-Net) to overcome these issues. TSS-Net utilizes two separate Siamese groups integrated with a registration module to learn differential features based on Siamese similarity comparison, and employs a feature complementary learning strategy, so that the model is able to simultaneously handle large deformations and generalize to unseen defects. Specifically, a pre-trained Registration Module (RM) is integrated into the triple-stream Siamese segmentation network to achieve end-to-end detection, which enhances the robustness to large deformations. Moreover, a group of Differential Feature Enhancement (DFE) modules are constructed to learn differential features at different scales, which forces the network to focus on changed features while ignores invariant features. Further, a feature complementary learning strategy is designed to fuse differential features at different scales, which can suppress artifacts during network reconstruction. Finally, a method is proposed to simulate printed labels with varying degrees of deformation and artificial defects. Extensive experimental results show that our proposed TSS-Net yields the best performance compared with the state-of-the-art method. Specifically, our proposed method achieves improvements of 4.20% on the F 1 score and 7.51% on the Intersection over Union (IoU).
- 27 May 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 34  (10) , 9910-9924
Multimodal Decoupled Distillation Graph Neural Network for Emotion Recognition in Conversation
Graph Neural Networks (GNNs) have attracted increasing attentions for multimodal Emotion Recognition in Conversation (ERC) due to their good performance in contextual understanding. However, most existing GNN-based methods suffer from two challenges: 1) How to explore and propagate appropriate information in a conversational graph. Typical GNNs in ERC neglect to mine the emotion commonality and discrepancy in the local neighborhood, leading to learn similar embbedings for connected nodes. However, the embeddings of these connected nodes are supposed to be distinguishable as they belong to different speakers with different emotions. 2) Most existing works apply simple concatenation or co-occurrence prior for modality combination, failing to fully capture the emotional information of multiple modalities in relationship modeling. In this paper, we propose a multimodal Decoupled Distillation Graph Neural Network (D 2 GNN) to address the above challenges. Specifically, D 2 GNN decouples the input features into emotion-aware and emotion-agnostic ones on the emotion category-level, aiming to capture emotion commonality and implicit emotion information, respectively. Moreover, we design a new message passing mechanism to separately propagate emotion-aware and -agnostic knowledge between nodes according to speaker dependency in two GNN-based modules, exploring the correlations of utterances and alleviating the similarities of embeddings. Furthermore, a multimodal distillation unit is performed to obtain the distinguishable embeddings by aggregating unimodal decoupled features. Experimental results on two ERC benchmarks demonstrate the superiority of the proposed model. Code is available at https://github.com/gityider/D2GNN.
- 27 May 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 34  (10) , 9910-9924
Multimodal Decoupled Distillation Graph Neural Network for Emotion Recognition in Conversation
Graph Neural Networks (GNNs) have attracted increasing attentions for multimodal Emotion Recognition in Conversation (ERC) due to their good performance in contextual understanding. However, most existing GNN-based methods suffer from two challenges: 1) How to explore and propagate appropriate information in a conversational graph. Typical GNNs in ERC neglect to mine the emotion commonality and discrepancy in the local neighborhood, leading to learn similar embbedings for connected nodes. However, the embeddings of these connected nodes are supposed to be distinguishable as they belong to different speakers with different emotions. 2) Most existing works apply simple concatenation or co-occurrence prior for modality combination, failing to fully capture the emotional information of multiple modalities in relationship modeling. In this paper, we propose a multimodal Decoupled Distillation Graph Neural Network (D 2 GNN) to address the above challenges. Specifically, D 2 GNN decouples the input features into emotion-aware and emotion-agnostic ones on the emotion category-level, aiming to capture emotion commonality and implicit emotion information, respectively. Moreover, we design a new message passing mechanism to separately propagate emotion-aware and -agnostic knowledge between nodes according to speaker dependency in two GNN-based modules, exploring the correlations of utterances and alleviating the similarities of embeddings. Furthermore, a multimodal distillation unit is performed to obtain the distinguishable embeddings by aggregating unimodal decoupled features. Experimental results on two ERC benchmarks demonstrate the superiority of the proposed model. Code is available at https://github.com/gityider/D2GNN.
- 23 May 2024
- Published by Elsevier in Knowledge-Based Systems
- Vol. 298
Multi-modal graph context extraction and consensus-aware learning for emotion recognition in conversation
- 23 May 2024
- Published by Elsevier in Knowledge-Based Systems
- Vol. 298
Multi-modal graph context extraction and consensus-aware learning for emotion recognition in conversation