Junda Zhao
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Junda Zhao has a total of 8 co-authors. Here are the top co-authors by number of shared publications:
- Eldan Cohen
University of Toronto, Department of Mechanical and Industrial Engineering, Toronto, Canada
4 shared publications - Shurui Zhou
University of Toronto, Department of Electrical and Computer Engineering, Toronto, Canada
2 shared publications Allen Chan
1 shared publicationArik Senderovich
1 shared publication- Eric Yu
Faculty of Information University of Toronto 140 St George St M5S 3G6 Toronto ON Canada
1 shared publication - Faria Khandaker
Department of Computer Science, Stamford University, Dhaka, Bangladesh
1 shared publication
Latest Publications
- Open Access
- 3 October 2026
- Published by Association for Computing Machinery (ACM)
- p. 12-14
Accurate Specification Recovery for Effective Test Generation
Large language model (LLM)-based unit-test generation has largely been studied under the assumption that the code under test is correct, using coverage or test-correctness metrics that do not directly measure defect detection. When the code under test is buggy, however, models can be guided toward assertions that encode the faulty behavior, producing tests that confirm rather than reveal defects. Our recent work quantified this misguidance effect and showed that a specification-based approach, which first recovers a behavioral specification of the focal method from its buggy implementation and then generates tests from that specification, can substantially mitigate it. Although this establishes a specification-recovery-first paradigm for LLM-based unit-test generation, the recovery step itself remains simple and its specifications are themselves fallible. This dissertation therefore aims to make specification recovery from imperfect code accurate, evidence-driven, and uncertainty-aware, so that it consistently yields effective bug-finding tests.
- 3 October 2026
- Published by Association for Computing Machinery (ACM)
- p. 12-14
Open AccessAccurate Specification Recovery for Effective Test Generation
Large language model (LLM)-based unit-test generation has largely been studied under the assumption that the code under test is correct, using coverage or test-correctness metrics that do not directly measure defect detection. When the code under test is buggy, however, models can be guided toward assertions that encode the faulty behavior, producing tests that confirm rather than reveal defects. Our recent work quantified this misguidance effect and showed that a specification-based approach, which first recovers a behavioral specification of the focal method from its buggy implementation and then generates tests from that specification, can substantially mitigate it. Although this establishes a specification-recovery-first paradigm for LLM-based unit-test generation, the recovery step itself remains simple and its specifications are themselves fallible. This dissertation therefore aims to make specification recovery from imperfect code accurate, evidence-driven, and uncertainty-aware, so that it consistently yields effective bug-finding tests.
- Open Access
- 1 October 2026
- Published by Association for Computing Machinery (ACM) in Proceedings of the ACM on Software Engineering
- Vol. 3 (ISSTA) , 2589-2612
Evaluating and Mitigating the Misguidance Effect of Buggy Code in LLM-Generated Unit Tests
While Large Language Models (LLMs) show great promise for automating unit test generation, recent studies suggest that the quality of generated tests can be negatively impacted when models are prompted with buggy code. This paper presents a new metric to quantitatively measure the "misguidance effect," a phenomenon where buggy code steers LLMs toward generating tests that validate its erroneous behavior rather than expose it. Our analysis reveals that prompting LLMs with buggy code has a severe, twofold impact: it significantly increases "misguided tests" that assert incorrect behavior while simultaneously suppressing the generation of effective, bug-finding tests. We further corroborate this effect from a model-internal perspective, showing that buggy code skews LLMs' preference toward tests that assert the same erroneous behavior. To counter this, we introduce and validate a specification-based unit test generation paradigm that replaces the code under test in the prompt with an LLM-generated specification docstring. Our results show that this paradigm effectively reduces misguided tests while substantially increasing effective tests, improves multi-round, feedback-driven test generation pipelines, and remains applicable to both buggy and bug-free code. Overall, these results suggest that specification-based prompting is a promising strategy for mitigating misguidance from buggy code in LLM-generated unit tests.
- 1 October 2026
- Published by Association for Computing Machinery (ACM) in Proceedings of the ACM on Software Engineering
- Vol. 3 (ISSTA) , 2589-2612
Open AccessEvaluating and Mitigating the Misguidance Effect of Buggy Code in LLM-Generated Unit Tests
While Large Language Models (LLMs) show great promise for automating unit test generation, recent studies suggest that the quality of generated tests can be negatively impacted when models are prompted with buggy code. This paper presents a new metric to quantitatively measure the "misguidance effect," a phenomenon where buggy code steers LLMs toward generating tests that validate its erroneous behavior rather than expose it. Our analysis reveals that prompting LLMs with buggy code has a severe, twofold impact: it significantly increases "misguided tests" that assert incorrect behavior while simultaneously suppressing the generation of effective, bug-finding tests. We further corroborate this effect from a model-internal perspective, showing that buggy code skews LLMs' preference toward tests that assert the same erroneous behavior. To counter this, we introduce and validate a specification-based unit test generation paradigm that replaces the code under test in the prompt with an LLM-generated specification docstring. Our results show that this paradigm effectively reduces misguided tests while substantially increasing effective tests, improves multi-round, feedback-driven test generation pipelines, and remains applicable to both buggy and bug-free code. Overall, these results suggest that specification-based prompting is a promising strategy for mitigating misguidance from buggy code in LLM-generated unit tests.
- Open Access
- 1 October 2026
- Published by Association for Computing Machinery (ACM) in Proceedings of the ACM on Software Engineering
- Vol. 3 (ISSTA) , 30-53
Do Coverage and Mutation Scores of LLM-Generated Test Suites Correlate with Their Effectiveness? (Replicability Study)
Recent advances in large language models (LLMs) have driven growing interest in using LLMs to automate test generation. Prior work commonly evaluates generated test suites using proxy metrics such as code coverage and mutation score. However, studies by Inozemtseva et al. and Papadakis et al. show that, for human-written tests, correlations among coverage, mutation, and real-bug detection can largely vanish once test suite size is controlled, raising concerns about the validity of evaluations based on proxy metrics. It also remains unclear whether these conclusions carry over to LLM-generated tests, given that prevailing LLM-based test-generation workflows differ substantially from traditional approaches. In this paper, we conduct a large-scale replication study of these two prior works using a wide range of test suites generated by a diverse set of LLMs, and re-examine the relationships among coverage, mutation, and real-bug detection effectiveness. Our findings diverge substantially from prior results. We show that the usefulness of coverage and mutation is highly context-dependent: in regression-style settings where the code provided to the LLM can be reasonably assumed bug-free, these metrics can provide meaningful signals when comparing across models; in another common scenario where the code-under-test may already be buggy and the goal is to expose the bug within the code-under-test, they no longer serve as reliable indicators. We also find little evidence that test suite size is a dominant confounder for correlations among coverage, mutation, and real-bug detection for LLM-generated tests. Based on these findings, we discuss how to interpret results from prior studies and provide actionable guidance for evaluating LLM-based test generation.
- 1 October 2026
- Published by Association for Computing Machinery (ACM) in Proceedings of the ACM on Software Engineering
- Vol. 3 (ISSTA) , 30-53
Open AccessDo Coverage and Mutation Scores of LLM-Generated Test Suites Correlate with Their Effectiveness? (Replicability Study)
Recent advances in large language models (LLMs) have driven growing interest in using LLMs to automate test generation. Prior work commonly evaluates generated test suites using proxy metrics such as code coverage and mutation score. However, studies by Inozemtseva et al. and Papadakis et al. show that, for human-written tests, correlations among coverage, mutation, and real-bug detection can largely vanish once test suite size is controlled, raising concerns about the validity of evaluations based on proxy metrics. It also remains unclear whether these conclusions carry over to LLM-generated tests, given that prevailing LLM-based test-generation workflows differ substantially from traditional approaches. In this paper, we conduct a large-scale replication study of these two prior works using a wide range of test suites generated by a diverse set of LLMs, and re-examine the relationships among coverage, mutation, and real-bug detection effectiveness. Our findings diverge substantially from prior results. We show that the usefulness of coverage and mutation is highly context-dependent: in regression-style settings where the code provided to the LLM can be reasonably assumed bug-free, these metrics can provide meaningful signals when comparing across models; in another common scenario where the code-under-test may already be buggy and the goal is to expose the bug within the code-under-test, they no longer serve as reliable indicators. We also find little evidence that test suite size is a dominant confounder for correlations among coverage, mutation, and real-bug detection for LLM-generated tests. Based on these findings, we discuss how to interpret results from prior studies and provide actionable guidance for evaluating LLM-based test generation.
- Open Access
- 21 May 2025
- Published by Elsevier in Journal of Systems and Software
- Vol. 229
Variational Prefix Tuning for diverse and accurate code summarization using pre-trained language models
- 21 May 2025
- Published by Elsevier in Journal of Systems and Software
- Vol. 229
Open AccessVariational Prefix Tuning for diverse and accurate code summarization using pre-trained language models
- 19 November 2023
- Published by Elsevier in Engineering Applications of Artificial Intelligence
- Vol. 128
Transformer models for mining intents and predicting activities from emails in knowledge-intensive processes
- 19 November 2023
- Published by Elsevier in Engineering Applications of Artificial Intelligence
- Vol. 128
Transformer models for mining intents and predicting activities from emails in knowledge-intensive processes