Yanzhou Li
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Yanzhou Li has a total of 9 co-authors. Here are the top co-authors by number of shared publications:
- Wei Ma
Singapore Management University, Singapore, Singapore
3 shared publications - Yang Liu
Department of Obstetrics and Gynecology, The Second Affiliated Hospital of Kunming Medical University, China
3 shared publications - Jian Zhang
Nanyang technological university, Singapore, Singapore
2 shared publications - Shangqing Liu
State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China
2 shared publications - Anran Li
School of Medicine, Yale University, New Haven, CT, USA
1 shared publication - Daya Guo
Sun Yat-Sen University, Guangzhou, China
1 shared publication
Latest Publications
- 16 September 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Software Engineering
- Vol. 51 (11) , 3088-3102
An Empirical Study of Exploring the Capabilities of Large Language Models in Code Learning
Since the advent of ChatGPT, large language models (LLMs) have attracted widespread attention from academia and industry. They have also brought significant changes to software engineering. However, until now, there has been a lack of comprehensive studies comparing LLMs with previous smaller code pre-trained models. To address this gap, we conduct a study in this paper to illustrate the performance of LLMs in different software engineering tasks. Specifically, we select three open-source large language models, CodeGen, LLaMA, and StarCoder, for the research targets, and our study is conducted from four aspects, including code syntax understanding, code semantic reasoning, encoding representation quality, and adaptation performance for different software engineering tasks to compare LLMs with previous code pre-trained models. Four aspects build on each other, forming important components of AI for Software Engineering. We conclude that: (1) Compared with previous smaller pre-trained models like CodeBERT, LLMs exhibit distinct trends in how they learn code syntax or semantics as the number of layers increases. Additionally, mastering code semantics proves to be more challenging, with semantic information usually learned in the final layers; (2) Causal decoder architecture with left-to-right attention masking does not perform well in zero-shot tasks; (3) For classification tasks, the mean vector representation generated by LLMs over a sequence tends to outperform the last token representation in the sequence; (4) Incorporating parameter-efficient fine-tuning techniques into LLMs for downstream tasks can help LLMs achieve better performance than previous code pre-trained models on code generation tasks but may not be optimal in some code understanding tasks; (5) LoRA emerges as a more effective PEFT technique for LLMs in downstream code-related tasks. We hope these findings will better guide future researchers in designing more powerful code models.
- 16 September 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Software Engineering
- Vol. 51 (11) , 3088-3102
An Empirical Study of Exploring the Capabilities of Large Language Models in Code Learning
Since the advent of ChatGPT, large language models (LLMs) have attracted widespread attention from academia and industry. They have also brought significant changes to software engineering. However, until now, there has been a lack of comprehensive studies comparing LLMs with previous smaller code pre-trained models. To address this gap, we conduct a study in this paper to illustrate the performance of LLMs in different software engineering tasks. Specifically, we select three open-source large language models, CodeGen, LLaMA, and StarCoder, for the research targets, and our study is conducted from four aspects, including code syntax understanding, code semantic reasoning, encoding representation quality, and adaptation performance for different software engineering tasks to compare LLMs with previous code pre-trained models. Four aspects build on each other, forming important components of AI for Software Engineering. We conclude that: (1) Compared with previous smaller pre-trained models like CodeBERT, LLMs exhibit distinct trends in how they learn code syntax or semantics as the number of layers increases. Additionally, mastering code semantics proves to be more challenging, with semantic information usually learned in the final layers; (2) Causal decoder architecture with left-to-right attention masking does not perform well in zero-shot tasks; (3) For classification tasks, the mean vector representation generated by LLMs over a sequence tends to outperform the last token representation in the sequence; (4) Incorporating parameter-efficient fine-tuning techniques into LLMs for downstream tasks can help LLMs achieve better performance than previous code pre-trained models on code generation tasks but may not be optimal in some code understanding tasks; (5) LoRA emerges as a more effective PEFT technique for LLMs in downstream code-related tasks. We hope these findings will better guide future researchers in designing more powerful code models.
- Open Access
- 1 July 2024
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Software Engineering and Methodology
- Vol. 33 (8)
Automated Commit Intelligence by Pre-training
GitHub commits, which record the code changes with natural language messages for description, play a critical role in software developers’ comprehension of software evolution. Due to their importance in software development, several learning-based works are conducted for GitHub commits, such as commit message generation and security patch identification. However, most existing works focus on customizing specialized neural networks for different tasks. Inspired by the superiority of code pre-trained models, which has confirmed their effectiveness across different downstream tasks, to promote the development of open-source software community, we first collect a large-scale commit benchmark including over 7.99 million commits across 7 programming languages. Based on this benchmark, we present CommitBART, a pre-trained encoder-decoder Transformer model for GitHub commits. The model is pre-trained by three categories (i.e., denoising objectives, cross-modal generation, and contrastive learning) for six pre-training tasks to learn commit fragment representations. Our model is evaluated on one understanding task and three generation tasks for commits. The comprehensive experiments on these tasks demonstrate that CommitBART significantly outperforms previous pre-trained works for code. Further analysis also reveals that each pre-training task enhances the model performance.
- 1 July 2024
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Software Engineering and Methodology
- Vol. 33 (8)
Open AccessAutomated Commit Intelligence by Pre-training
GitHub commits, which record the code changes with natural language messages for description, play a critical role in software developers’ comprehension of software evolution. Due to their importance in software development, several learning-based works are conducted for GitHub commits, such as commit message generation and security patch identification. However, most existing works focus on customizing specialized neural networks for different tasks. Inspired by the superiority of code pre-trained models, which has confirmed their effectiveness across different downstream tasks, to promote the development of open-source software community, we first collect a large-scale commit benchmark including over 7.99 million commits across 7 programming languages. Based on this benchmark, we present CommitBART, a pre-trained encoder-decoder Transformer model for GitHub commits. The model is pre-trained by three categories (i.e., denoising objectives, cross-modal generation, and contrastive learning) for six pre-training tasks to learn commit fragment representations. Our model is evaluated on one understanding task and three generation tasks for commits. The comprehensive experiments on these tasks demonstrate that CommitBART significantly outperforms previous pre-trained works for code. Further analysis also reveals that each pre-training task enhances the model performance.
- Open Access
- 12 April 2024
- Published by Association for Computing Machinery (ACM)
- p. 1-12
An Empirical Study on Noisy Label Learning for Program Understanding
Recently, deep learning models have been widely applied in program understanding tasks, and these models achieve state-of-the-art results on many benchmark datasets. A major challenge of deep learning for program understanding is that the effectiveness of these approaches depends on the quality of their datasets, and these datasets often contain noisy data samples. A typical kind of noise in program understanding datasets is label noise, which means that the target outputs for some inputs are incorrect. Researchers have proposed various approaches to alleviate the negative impact of noisy labels, and formed a new research topic: noisy label learning (NLL). In this paper, we conduct an empirical study on the effectiveness of noisy label learning on deep learning for program understanding datasets. We evaluate various NLL approaches and deep learning models on three tasks: program classification, vulnerability detection, and code summarization. From the evaluation results, we come to the following findings: 1) small trained-from-scratch models are prone to label noises in program understanding, while large pre-trained models are highly robust against them. 2) NLL approaches significantly improve the program classification accuracies for small models on noisy training sets, but they only slightly benefit large pre-trained models in classification accuracies. 3) NLL can effectively detect synthetic noises in program understanding, but struggle in detecting real-world noises. We believe our findings can provide insights on the abilities of NLL in program understanding, and shed light on future works in tackling noises in software engineering datasets. We have released our code at https://github.com/jacobwwh/noise_SE.
- 12 April 2024
- Published by Association for Computing Machinery (ACM)
- p. 1-12
Open AccessAn Empirical Study on Noisy Label Learning for Program Understanding
Recently, deep learning models have been widely applied in program understanding tasks, and these models achieve state-of-the-art results on many benchmark datasets. A major challenge of deep learning for program understanding is that the effectiveness of these approaches depends on the quality of their datasets, and these datasets often contain noisy data samples. A typical kind of noise in program understanding datasets is label noise, which means that the target outputs for some inputs are incorrect. Researchers have proposed various approaches to alleviate the negative impact of noisy labels, and formed a new research topic: noisy label learning (NLL). In this paper, we conduct an empirical study on the effectiveness of noisy label learning on deep learning for program understanding datasets. We evaluate various NLL approaches and deep learning models on three tasks: program classification, vulnerability detection, and code summarization. From the evaluation results, we come to the following findings: 1) small trained-from-scratch models are prone to label noises in program understanding, while large pre-trained models are highly robust against them. 2) NLL approaches significantly improve the program classification accuracies for small models on noisy training sets, but they only slightly benefit large pre-trained models in classification accuracies. 3) NLL can effectively detect synthetic noises in program understanding, but struggle in detecting real-world noises. We believe our findings can provide insights on the abilities of NLL in program understanding, and shed light on future works in tackling noises in software engineering datasets. We have released our code at https://github.com/jacobwwh/noise_SE.