Wentao Zhang
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Wentao Zhang has a total of 26 co-authors. Here are the top co-authors by number of shared publications:
- Ruixuan Wang
School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, China
6 shared publications - Wei-Shi Zheng
Sun Yat-sen University
4 shared publications - Junxi Chen
School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, China
2 shared publications - Qicheng Lao
Beijing University of Posts and Telecommunications
2 shared publications - Emanuele Trucco
VAMPIRE project, Computing School of Science and Engineering, University of Dundee, Dundee, UK
1 shared publication Guangxing Wu
School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, China
1 shared publication
Latest Publications
- Open Access
- 17 April 2026
- Published by Elsevier in Engineering
Learning Conceptual Text Prompts from Visual Regions of Interest for Medical Image Segmentation
- 17 April 2026
- Published by Elsevier in Engineering
Open AccessLearning Conceptual Text Prompts from Visual Regions of Interest for Medical Image Segmentation
- 9 December 2025
- Published by Elsevier in Neural Networks
- Vol. 196, 108422
Dual-modality adaptation in vision-language models for continual learning
- 9 December 2025
- Published by Elsevier in Neural Networks
- Vol. 196, 108422
Dual-modality adaptation in vision-language models for continual learning
- 21 February 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 27 (15209210) , 5426-5438
Visual Class Incremental Learning With Textual Priors Guidance Based on an Adapted Vision-Language Model
An ideal artificial intelligence (AI) system should have the capability to continually learn like humans. However, when learning new knowledge, AI systems often suffer from catastrophic forgetting of old knowledge. Although many continual learning methods have been proposed, they often ignore the issue of misclassifying similar classes and make insufficient use of textual priors of visual classes to improve continual learning performance. In this study, we propose a continual learning framework based on a pre-trained vision-language model (VLM) that does not require storing old class data. This framework utilizes parameter-efficient fine-tuning of the VLM's text encoder for constructing a shared and consistent semantic textual space throughout the continual learning process. The textual priors of visual classes are encoded by the adapted VLM's text encoder to generate discriminative semantic representations, which are then used to guide the learning of visual classes. Additionally, fake out-of-distribution (OOD) images constructed from each training image further assist in the learning of visual classes. Extensive empirical evaluations on three natural datasets and one medical dataset demonstrate the superiority of the proposed framework. The source code is available at https://openi.pcl.ac.cn/OpenMedIA/CIL_Adapterd_VLM.
- 21 February 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 27 (15209210) , 5426-5438
Visual Class Incremental Learning With Textual Priors Guidance Based on an Adapted Vision-Language Model
An ideal artificial intelligence (AI) system should have the capability to continually learn like humans. However, when learning new knowledge, AI systems often suffer from catastrophic forgetting of old knowledge. Although many continual learning methods have been proposed, they often ignore the issue of misclassifying similar classes and make insufficient use of textual priors of visual classes to improve continual learning performance. In this study, we propose a continual learning framework based on a pre-trained vision-language model (VLM) that does not require storing old class data. This framework utilizes parameter-efficient fine-tuning of the VLM's text encoder for constructing a shared and consistent semantic textual space throughout the continual learning process. The textual priors of visual classes are encoded by the adapted VLM's text encoder to generate discriminative semantic representations, which are then used to guide the learning of visual classes. Additionally, fake out-of-distribution (OOD) images constructed from each training image further assist in the learning of visual classes. Extensive empirical evaluations on three natural datasets and one medical dataset demonstrate the superiority of the proposed framework. The source code is available at https://openi.pcl.ac.cn/OpenMedIA/CIL_Adapterd_VLM.
- 13 February 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 35 (7) , 7143-7154
DAT: Dual-Branch Adapter-Tuning for Few-Shot Recognition
Parameter-Efficient Fine-Tuning methods based on vision-language models (such as CLIP) for few-shot learning have recently received considerable attention. However, previous works only fine-tune either the image or text branch, breaking the alignment of the original two branches, meanwhile fine-tuning both branches of the CLIP would inevitably introduce more trainable parameters and likely cause more severe over-fitting due to the limited training data. In this study, we propose a novel Dual-branch Adapter-Tuning framework (DAT), which collaboratively trains the visual adapter and textual adapter added to the two branches of the original CLIP with multiple consistency constraints. By effectively utilizing the semantically detailed class-specific prompts and outputs of the original CLIP to guide the fine-tuning of both branches, our method gains exceptional adaptation ability to the downstream few-shot learning tasks and alleviates the over-fitting issue, meanwhile maximally preserving the generalization ability of the original CLIP model. Our proposed framework has achieved superior performance on diverse datasets under various few-shot learning settings compared to the existing approaches. The source code is available at https://github.com/SandyXi/DAT.
- 13 February 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 35 (7) , 7143-7154
DAT: Dual-Branch Adapter-Tuning for Few-Shot Recognition
Parameter-Efficient Fine-Tuning methods based on vision-language models (such as CLIP) for few-shot learning have recently received considerable attention. However, previous works only fine-tune either the image or text branch, breaking the alignment of the original two branches, meanwhile fine-tuning both branches of the CLIP would inevitably introduce more trainable parameters and likely cause more severe over-fitting due to the limited training data. In this study, we propose a novel Dual-branch Adapter-Tuning framework (DAT), which collaboratively trains the visual adapter and textual adapter added to the two branches of the original CLIP with multiple consistency constraints. By effectively utilizing the semantically detailed class-specific prompts and outputs of the original CLIP to guide the fine-tuning of both branches, our method gains exceptional adaptation ability to the downstream few-shot learning tasks and alleviates the over-fitting issue, meanwhile maximally preserving the generalization ability of the original CLIP model. Our proposed framework has achieved superior performance on diverse datasets under various few-shot learning settings compared to the existing approaches. The source code is available at https://github.com/SandyXi/DAT.
- 23 August 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 34 (12) , 13152-13163
Continual Learning of Image Classes With Language Guidance From a Vision-Language Model
Current deep learning models often catastrophically forget the knowledge of old classes when continually learning new ones. State-of-the-art approaches to continual learning of image classes often require retaining a small subset of old data to partly alleviate the catastrophic forgetting issue, and their performance would be degraded sharply when no old data can be stored due to privacy or safety concerns. In this study, inspired by human learning of visual knowledge with the effective help of language, we propose a novel continual learning framework based on a pre-trained vision-language model (VLM) without retaining any old data. Rich prior knowledge of each new image class is effectively encoded by the frozen text encoder of the VLM, which is then used to guide the learning of new image classes. The output space of the frozen text encoder is unchanged over the whole process of continual learning, through which image representations of different classes become comparable during model inference even when the image classes are learned at different times. Extensive empirical evaluations on multiple image classification datasets under various settings confirm the superior performance of our method over existing ones. The source code will be released publicly. The source code is available at https://github.com/Fatflower/CIL LG VLM/.
- 23 August 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 34 (12) , 13152-13163
Continual Learning of Image Classes With Language Guidance From a Vision-Language Model
Current deep learning models often catastrophically forget the knowledge of old classes when continually learning new ones. State-of-the-art approaches to continual learning of image classes often require retaining a small subset of old data to partly alleviate the catastrophic forgetting issue, and their performance would be degraded sharply when no old data can be stored due to privacy or safety concerns. In this study, inspired by human learning of visual knowledge with the effective help of language, we propose a novel continual learning framework based on a pre-trained vision-language model (VLM) without retaining any old data. Rich prior knowledge of each new image class is effectively encoded by the frozen text encoder of the VLM, which is then used to guide the learning of new image classes. The output space of the frozen text encoder is unchanged over the whole process of continual learning, through which image representations of different classes become comparable during model inference even when the image classes are learned at different times. Extensive empirical evaluations on multiple image classification datasets under various settings confirm the superior performance of our method over existing ones. The source code will be released publicly. The source code is available at https://github.com/Fatflower/CIL LG VLM/.
- Open Access
- 6 December 2023
- Published by Association for Computing Machinery (ACM)
- p. 1-7
Feature Adaptation with CLIP for Few-shot Classification
Large Vision-Language models such as CLIP have demonstrated impressive capabilities in zero-shot recognition. To apply CLIP to few-shot classification tasks, several methods have been proposed based on CLIP, achieving significant improvements. However, these methods either insufficiently leverage CLIP’s prior knowledge during training or neglect the impact of feature adaptation. In this paper, we propose FAR, a novel approach that balances distribution-altered Feature Adaptation with pRior knowledge of CLIP to further improve the performance of CLIP in few-shot classification tasks. Firstly, we introduce an adapter that enhances the effectiveness of CLIP adaptation by amplifying the differences between the fine-tuned CLIP features and the original CLIP features. Secondly, we leverage the prior knowledge of CLIP to mitigate the risk of overfitting. Through this framework, a good trade-off between feature adaptation and preserving prior knowledge is achieved, enabling effective utilization of both components to enhance performance on downstream tasks. We evaluate our method on over 10 datasets for classification, and our approach consistently outperforms existing methods, demonstrating its effectiveness and robustness.
- 6 December 2023
- Published by Association for Computing Machinery (ACM)
- p. 1-7
Open AccessFeature Adaptation with CLIP for Few-shot Classification
Large Vision-Language models such as CLIP have demonstrated impressive capabilities in zero-shot recognition. To apply CLIP to few-shot classification tasks, several methods have been proposed based on CLIP, achieving significant improvements. However, these methods either insufficiently leverage CLIP’s prior knowledge during training or neglect the impact of feature adaptation. In this paper, we propose FAR, a novel approach that balances distribution-altered Feature Adaptation with pRior knowledge of CLIP to further improve the performance of CLIP in few-shot classification tasks. Firstly, we introduce an adapter that enhances the effectiveness of CLIP adaptation by amplifying the differences between the fine-tuned CLIP features and the original CLIP features. Secondly, we leverage the prior knowledge of CLIP to mitigate the risk of overfitting. Through this framework, a good trade-off between feature adaptation and preserving prior knowledge is achieved, enabling effective utilization of both components to enhance performance on downstream tasks. We evaluate our method on over 10 datasets for classification, and our approach consistently outperforms existing methods, demonstrating its effectiveness and robustness.
- Open Access
- 18 November 2023
- Published by Association for Computing Machinery (ACM)
- p. 10-14
Local Adversarial Attacks for Understanding Model Decisions
Recently, deep learning models have demonstrated outstanding performance, but the lack of interpretability makes it difficult to apply them to high-risk tasks. Many existing post-hoc explanation methods have been proposed to explain specific model predictions, but they mostly focus on which locations influence the model decision. In other words, they only provide the location information of discriminative regions but cannot provide the visual differences of the relevant categories in these regions. In this paper, a simple yet effective approach is proposed to explain specific model prediction, which uses visual differences between different categories to provide more intuitive explanations. The basic idea is to use adversarial attacks to transform discriminative features in local regions of an image into features of another class, thereby achieving an explanation of the model's predictions. In this way, the location difference between the original class and the target class in the image will be localized, and an intuitive explanation will be provided through visual changes within the region. Extensive evaluation demonstrates that the proposed method can accurately and adaptively locate discriminative regions in images, and provide users with an intuitive explanation of the classification basis for model predictions through visualization results.
- 18 November 2023
- Published by Association for Computing Machinery (ACM)
- p. 10-14
Open AccessLocal Adversarial Attacks for Understanding Model Decisions
Recently, deep learning models have demonstrated outstanding performance, but the lack of interpretability makes it difficult to apply them to high-risk tasks. Many existing post-hoc explanation methods have been proposed to explain specific model predictions, but they mostly focus on which locations influence the model decision. In other words, they only provide the location information of discriminative regions but cannot provide the visual differences of the relevant categories in these regions. In this paper, a simple yet effective approach is proposed to explain specific model prediction, which uses visual differences between different categories to provide more intuitive explanations. The basic idea is to use adversarial attacks to transform discriminative features in local regions of an image into features of another class, thereby achieving an explanation of the model's predictions. In this way, the location difference between the original class and the target class in the image will be localized, and an intuitive explanation will be provided through visual changes within the region. Extensive evaluation demonstrates that the proposed method can accurately and adaptively locate discriminative regions in images, and provide users with an intuitive explanation of the classification basis for model predictions through visualization results.