Yujia Zhang
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Yujia Zhang has a total of 25 co-authors. Here are the top co-authors by number of shared publications:
- Xiaoguang Zhao
State Key Laboratory of Multimodal Artificial Intelligence Systems Institute of Automation Chinese Academy of Sciences
7 shared publications - Michael Kampffmeyer
UiT The Arctic University of Norway
6 shared publications Eric P. Xing
5 shared publicationsXiaodan Liang
5 shared publicationsXiaoguang Zhao
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China
5 shared publicationsDingwen Zhang
4 shared publications
Latest Publications
- 11 May 2026
- Published by Elsevier in Neural Networks
- Vol. 202, 109092
ProPy : Building interactive and efficient prompt pyramids upon CLIP for partially relevant video retrieval
- 11 May 2026
- Published by Elsevier in Neural Networks
- Vol. 202, 109092
ProPy : Building interactive and efficient prompt pyramids upon CLIP for partially relevant video retrieval
- Open Access
ProPy: Building Interactive Prompt Pyramids upon CLIP for Partially Relevant Video Retrieval
Partially Relevant Video Retrieval (PRVR) is a practical yet challenging task that involves retrieving videos based on queries relevant to only specific segments. While existing works follow the paradigm of developing models to process unimodal features, powerful pretrained vision-language models like CLIP remain underexplored in this field. To bridge this gap, we propose ProPy, a model with systematic architectural adaption of CLIP specifically designed for PRVR. Drawing insights from the semantic relevance of multi-granularity events, ProPy introduces two key innovations: (1) A Prompt Pyramid structure that organizes event prompts to capture semantics at multiple granularity levels, and (2) An Ancestor-Descendant Interaction Mechanism built on the pyramid that enables dynamic semantic interaction among events. With these designs, ProPy achieves SOTA performance on three public datasets, outperforming previous models by significant margins. Code is available at https://github.com/BUAAPY/ProPy.
Open AccessProPy: Building Interactive Prompt Pyramids upon CLIP for Partially Relevant Video Retrieval
Partially Relevant Video Retrieval (PRVR) is a practical yet challenging task that involves retrieving videos based on queries relevant to only specific segments. While existing works follow the paradigm of developing models to process unimodal features, powerful pretrained vision-language models like CLIP remain underexplored in this field. To bridge this gap, we propose ProPy, a model with systematic architectural adaption of CLIP specifically designed for PRVR. Drawing insights from the semantic relevance of multi-granularity events, ProPy introduces two key innovations: (1) A Prompt Pyramid structure that organizes event prompts to capture semantics at multiple granularity levels, and (2) An Ancestor-Descendant Interaction Mechanism built on the pyramid that enables dynamic semantic interaction among events. With these designs, ProPy achieves SOTA performance on three public datasets, outperforming previous models by significant margins. Code is available at https://github.com/BUAAPY/ProPy.
- Open Access
- 1 January 2025
- Published by Association for Computational Linguistics (ACL)
- p. 519-533
ProPy: Building Interactive Prompt Pyramids upon CLIP for Partially Relevant Video Retrieval
Yi Pan, Yujia Zhang, Michael Kampffmeyer, Xiaoguang Zhao. Findings of the Association for Computational Linguistics: EMNLP 2025. 2025.
- 1 January 2025
- Published by Association for Computational Linguistics (ACL)
- p. 519-533
Open AccessProPy: Building Interactive Prompt Pyramids upon CLIP for Partially Relevant Video Retrieval
Yi Pan, Yujia Zhang, Michael Kampffmeyer, Xiaoguang Zhao. Findings of the Association for Computational Linguistics: EMNLP 2025. 2025.
- 14 December 2024
- Published by Springer Nature in Multimedia Systems
- Vol. 31 (1) , 1-17
HierGAT: hierarchical spatial-temporal network with graph and transformer for video HOI detection
- 14 December 2024
- Published by Springer Nature in Multimedia Systems
- Vol. 31 (1) , 1-17
HierGAT: hierarchical spatial-temporal network with graph and transformer for video HOI detection
- 16 November 2024
- Published by Elsevier in Neurocomputing
- Vol. 616
Prompt-guided bidirectional deep fusion network for referring image segmentation
- 16 November 2024
- Published by Elsevier in Neurocomputing
- Vol. 616
Prompt-guided bidirectional deep fusion network for referring image segmentation
- 2 May 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 33 (10577149) , 3256-3270
Multi-Stage Image-Language Cross-Generative Fusion Network for Video-Based Referring Expression Comprehension
Video-based referring expression comprehension is a challenging task that requires locating the referred object in each video frame of a given video. While many existing approaches treat this task as an object-tracking problem, their performance is heavily reliant on the quality of the tracking templates. Furthermore, when there is not enough annotation data to assist in template selection, the tracking may fail. Other approaches are based on object detection, but they often use only one adjacent frame of the key frame for feature learning, which limits their ability to establish the relationship between different frames. In addition, improving the fusion of features from multiple frames and referring expressions to effectively locate the referents remains an open problem. To address these issues, we propose a novel approach called the Multi-Stage Image-Language Cross-Generative Fusion Network (MILCGF-Net), which is based on one-stage object detection. Our approach includes a Frame Dense Feature Aggregation module for dense feature learning of adjacent time sequences. Additionally, we propose an Image-Language Cross-Generative Fusion module as the main body of multi-stage learning to generate cross-modal features by calculating the similarity between video and expression, and then refining and fusing the generated features. To further enhance the cross-modal feature generation capability of our model, we introduce a consistency loss that constrains the image-language similarity and language-image similarity matrices during feature generation. We evaluate our proposed approach on three public datasets and demonstrate its effectiveness through comprehensive experimental results.
- 2 May 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 33 (10577149) , 3256-3270
Multi-Stage Image-Language Cross-Generative Fusion Network for Video-Based Referring Expression Comprehension
Video-based referring expression comprehension is a challenging task that requires locating the referred object in each video frame of a given video. While many existing approaches treat this task as an object-tracking problem, their performance is heavily reliant on the quality of the tracking templates. Furthermore, when there is not enough annotation data to assist in template selection, the tracking may fail. Other approaches are based on object detection, but they often use only one adjacent frame of the key frame for feature learning, which limits their ability to establish the relationship between different frames. In addition, improving the fusion of features from multiple frames and referring expressions to effectively locate the referents remains an open problem. To address these issues, we propose a novel approach called the Multi-Stage Image-Language Cross-Generative Fusion Network (MILCGF-Net), which is based on one-stage object detection. Our approach includes a Frame Dense Feature Aggregation module for dense feature learning of adjacent time sequences. Additionally, we propose an Image-Language Cross-Generative Fusion module as the main body of multi-stage learning to generate cross-modal features by calculating the similarity between video and expression, and then refining and fusing the generated features. To further enhance the cross-modal feature generation capability of our model, we introduce a consistency loss that constrains the image-language similarity and language-image similarity matrices during feature generation. We evaluate our proposed approach on three public datasets and demonstrate its effectiveness through comprehensive experimental results.
- 30 June 2023
- Published by Springer Nature in Applied Intelligence
- Vol. 53 (19) , 22565-22581
Visual enhanced hierarchical network for sentence-based video thumbnail generation
- 30 June 2023
- Published by Springer Nature in Applied Intelligence
- Vol. 53 (19) , 22565-22581
Visual enhanced hierarchical network for sentence-based video thumbnail generation
- 22 December 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Intelligent Transportation Systems
- Vol. 23 (9) , 14820-14832
Beyond Crack: Fine-Grained Pavement Defect Segmentation Using Three-Stream Neural Networks
Pavement defect segmentation is a fundamental task in the field of transport infrastructure inspection. Existing methods mainly focus on detection/segmentation for long and thin cracks. However, there are many other types of defects with various sizes and shapes that are also essential to segment, which brings more challenges toward detailed road inspection. To address the above problems and provide a more comprehensive understanding of the overall road conditions, we propose a three-stream neural network that combines spatial, contextual and boundary information for fine-grained defect segmentation. Specifically, the spatial stream captures rich low-level spatial features. The contextual stream utilizes an attention mechanism and models high-level contextual relationships over local features. To further refine the segmentation results, the boundary stream encodes detailed boundaries using a global gated convolution and generates additional boundary maps. By combining the above different information, our model can effectively produce pixel-wise predictions for fine-grained road inspection. The network is trained using a dual-task loss in an end-to-end manner, and experiments were performed on three newly collected datasets, i.e., a fine-grained defect dataset and two crack datasets, which shows that the proposed method achieves favorable segmentation results on complex multi-class defects, and is also able to segment single-class cracks. Specifically, on the fine-grained dataset, it achieved state-of-the-art performance over other competing baselines (mPA of 0.54, mIoU of 0.38, Mic_F of 0.78 and Mac_F of 0.65), where each image is resized to 512 $\times 512$ and the processing speed is 21 FPS on average.
- 22 December 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Intelligent Transportation Systems
- Vol. 23 (9) , 14820-14832
Beyond Crack: Fine-Grained Pavement Defect Segmentation Using Three-Stream Neural Networks
Pavement defect segmentation is a fundamental task in the field of transport infrastructure inspection. Existing methods mainly focus on detection/segmentation for long and thin cracks. However, there are many other types of defects with various sizes and shapes that are also essential to segment, which brings more challenges toward detailed road inspection. To address the above problems and provide a more comprehensive understanding of the overall road conditions, we propose a three-stream neural network that combines spatial, contextual and boundary information for fine-grained defect segmentation. Specifically, the spatial stream captures rich low-level spatial features. The contextual stream utilizes an attention mechanism and models high-level contextual relationships over local features. To further refine the segmentation results, the boundary stream encodes detailed boundaries using a global gated convolution and generates additional boundary maps. By combining the above different information, our model can effectively produce pixel-wise predictions for fine-grained road inspection. The network is trained using a dual-task loss in an end-to-end manner, and experiments were performed on three newly collected datasets, i.e., a fine-grained defect dataset and two crack datasets, which shows that the proposed method achieves favorable segmentation results on complex multi-class defects, and is also able to segment single-class cracks. Specifically, on the fine-grained dataset, it achieved state-of-the-art performance over other competing baselines (mPA of 0.54, mIoU of 0.38, Mic_F of 0.78 and Mac_F of 0.65), where each image is resized to 512 $\times 512$ and the processing speed is 21 FPS on average.
- 1 November 2021
- Published by Springer Nature in Applied Intelligence
- Vol. 52 (8) , 8616-8634
Generalized zero-shot emotion recognition from body gestures
- 1 November 2021
- Published by Springer Nature in Applied Intelligence
- Vol. 52 (8) , 8616-8634
Generalized zero-shot emotion recognition from body gestures
- Open Access
TB-Net: A Three-Stream Boundary-Aware Network for Fine-Grained Pavement Disease Segmentation
Regular pavement inspection plays a significant role in road maintenance for safety assurance. Existing methods mainly address the tasks of crack detection and segmentation that are only tailored for long-thin crack disease. However, there are many other types of diseases with a wider variety of sizes and patterns that are also essential to segment in practice, bringing more challenges towards fine-grained pavement inspection. In this paper, our goal is not only to automatically segment cracks, but also to segment other complex pavement diseases as well as typical landmarks (markings, runway lights, etc.) and commonly seen water/oil stains in a single model. To this end, we propose a three-stream boundary-aware network (TB-Net). It consists of three streams fusing the low-level spatial and the high-level contextual representations as well as the detailed boundary information. Specifically, the spatial stream captures rich spatial features. The context stream, where an attention mechanism is utilized, models the contextual relationships over local features. The boundary stream learns detailed boundaries using a global-gated convolution to further refine the segmentation outputs. The network is trained using a dual-task loss in an end-to-end manner, and experiments on a newly collected fine-grained pavement disease dataset show the effectiveness of our TB-Net.
Open AccessTB-Net: A Three-Stream Boundary-Aware Network for Fine-Grained Pavement Disease Segmentation
Regular pavement inspection plays a significant role in road maintenance for safety assurance. Existing methods mainly address the tasks of crack detection and segmentation that are only tailored for long-thin crack disease. However, there are many other types of diseases with a wider variety of sizes and patterns that are also essential to segment in practice, bringing more challenges towards fine-grained pavement inspection. In this paper, our goal is not only to automatically segment cracks, but also to segment other complex pavement diseases as well as typical landmarks (markings, runway lights, etc.) and commonly seen water/oil stains in a single model. To this end, we propose a three-stream boundary-aware network (TB-Net). It consists of three streams fusing the low-level spatial and the high-level contextual representations as well as the detailed boundary information. Specifically, the spatial stream captures rich spatial features. The context stream, where an attention mechanism is utilized, models the contextual relationships over local features. The boundary stream learns detailed boundaries using a global-gated convolution to further refine the segmentation outputs. The network is trained using a dual-task loss in an end-to-end manner, and experiments on a newly collected fine-grained pavement disease dataset show the effectiveness of our TB-Net.