Jiajun Jiang
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Jiajun Jiang has a total of 58 co-authors. Here are the top co-authors by number of shared publications:
- Junjie Chen
College of Intelligence and Computing, Tianjin University, China
14 shared publications - Fengjie Li
Tianjin University, School of Computer Software, Tianjin, China
4 shared publications - Fengjie Li
Tianjin University Tianjin China
4 shared publications - Hanmo You
Tianjin University, Tianjin, China
3 shared publications - Hongyu Zhang
Chongqing University, Chongqing, China
3 shared publications - Yingfei Xiong
Key Laboratory of High Confidence Software Technologies (Peking University), Ministry of Education, School of Computer Science, Peking University, Beijing, China
3 shared publications
Latest Publications
- Open Access
- 8 October 2026
- Published by Association for Computing Machinery (ACM)
- p. 9-16
AnchorRepair: A Reference-Guided Dual-Agent Framework for Code Translation Repair
Automated code translators can now produce repository-scale translations from Java to Python, yet a substantial fraction of translated functions remain semantically incorrect. Repairing such defects differs from traditional automated program repair (APR) because the correctness specification is based on the source language and no target-language test suite exists to guide fault localization. We observe that the original Java code is the most authoritative specification for repair and propose AnchorRepair, a dual-agent framework that treats the Java source as first-class diagnostic evidence. The Diagnosis Agent operates in a ReAct-style loop and autonomously invokes seven cross-lingual analysis tools, producing a structured root-cause diagnosis by contrasting the Java specification against the Python translation across control flow, data flow, and call graph dimensions. A Patch Agent then generates a corrected Python function conditioned on this diagnosis and the retrieved Java context. We evaluated AnchorRepair on 82 real translation defects from four Apache Commons projects. With DeepSeek-V3.2 as the backend, AnchorRepair repairs 40 defects (48.8%), a 12-point absolute improvement over the LLM-Only baseline (28/82) and substantially outperforming TransAgent (17/82) and D4C (21/82). The framework generalizes across LLM backends without modification and achieves repair coverage comparable to a general-purpose coding agent at less than half the cost. Our source code and all experimental results are publicly available at https://github.com/JJS-TJ/AnchorRepair.
- 8 October 2026
- Published by Association for Computing Machinery (ACM)
- p. 9-16
Open AccessAnchorRepair: A Reference-Guided Dual-Agent Framework for Code Translation Repair
Automated code translators can now produce repository-scale translations from Java to Python, yet a substantial fraction of translated functions remain semantically incorrect. Repairing such defects differs from traditional automated program repair (APR) because the correctness specification is based on the source language and no target-language test suite exists to guide fault localization. We observe that the original Java code is the most authoritative specification for repair and propose AnchorRepair, a dual-agent framework that treats the Java source as first-class diagnostic evidence. The Diagnosis Agent operates in a ReAct-style loop and autonomously invokes seven cross-lingual analysis tools, producing a structured root-cause diagnosis by contrasting the Java specification against the Python translation across control flow, data flow, and call graph dimensions. A Patch Agent then generates a corrected Python function conditioned on this diagnosis and the retrieved Java context. We evaluated AnchorRepair on 82 real translation defects from four Apache Commons projects. With DeepSeek-V3.2 as the backend, AnchorRepair repairs 40 defects (48.8%), a 12-point absolute improvement over the LLM-Only baseline (28/82) and substantially outperforming TransAgent (17/82) and D4C (21/82). The framework generalizes across LLM backends without modification and achieves repair coverage comparable to a general-purpose coding agent at less than half the cost. Our source code and all experimental results are publicly available at https://github.com/JJS-TJ/AnchorRepair.
- Open Access
- 25 June 2026
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Software Engineering and Methodology
Intent-Aware Defect Pattern Extraction from Singular Examples
As software systems grow in complexity, particularly in safety-critical domains, detecting defects early is crucial to prevent severe failures. Among existing automated defect detection methods, rule-based systems excel at catching well-understood mistakes but are hard to generalize beyond their predefined rules. To address this challenge, we introduce DslGen, a novel approach that automatically extracts high-quality defect patterns from singular defect repair examples by leveraging large language models (LLMs). Specifically, it first infers the semantic intent of a defect repair for capturing the defect’s essential characteristics via LLM-powered analysis, then generates precise constraint rules to define the defect pattern. Unlike prior methods, DslGen does not require numerous examples or rigid abstraction schemes, enabling robust generalization even from a single fix. To evaluate the effectiveness of DslGen, we evaluated it in two distinct scenarios: (1) controlled experiments and (2) a simulated real-world defect detection pipeline. In the controlled setting, results show that DslGen outperforms all baselines, achieving a precision up to 81.1% and a recall of over 73.0%. In the simulated scenario, DslGen also performed best with an average improvement of 425.1% over state-of-the-art competitors (including GPT-4o). A user study with 15 experienced developers further confirmed DslGen’s practicality, with participants praising the readability and generalizability of its generated DSLs and expressing strong adoption intent.
- 25 June 2026
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Software Engineering and Methodology
Open AccessIntent-Aware Defect Pattern Extraction from Singular Examples
As software systems grow in complexity, particularly in safety-critical domains, detecting defects early is crucial to prevent severe failures. Among existing automated defect detection methods, rule-based systems excel at catching well-understood mistakes but are hard to generalize beyond their predefined rules. To address this challenge, we introduce DslGen, a novel approach that automatically extracts high-quality defect patterns from singular defect repair examples by leveraging large language models (LLMs). Specifically, it first infers the semantic intent of a defect repair for capturing the defect’s essential characteristics via LLM-powered analysis, then generates precise constraint rules to define the defect pattern. Unlike prior methods, DslGen does not require numerous examples or rigid abstraction schemes, enabling robust generalization even from a single fix. To evaluate the effectiveness of DslGen, we evaluated it in two distinct scenarios: (1) controlled experiments and (2) a simulated real-world defect detection pipeline. In the controlled setting, results show that DslGen outperforms all baselines, achieving a precision up to 81.1% and a recall of over 73.0%. In the simulated scenario, DslGen also performed best with an average improvement of 425.1% over state-of-the-art competitors (including GPT-4o). A user study with 15 experienced developers further confirmed DslGen’s practicality, with participants praising the readability and generalizability of its generated DSLs and expressing strong adoption intent.
- Open Access
- 12 April 2026
- Published by Association for Computing Machinery (ACM)
- p. 717-729
Towards Better Linux Kernel Fault Localization: Leveraging Contrastive Reasoning and Hierarchical Context Analysis
Debugging the Linux kernel remains a formidable challenge due to its vast codebase, complex architecture, and low-level programming intricacies. Effective fault localization (FL) is thus essential for efficient kernel debugging and maintenance. While existing FL techniques (both traditional and LLM-based) have shown promise in general-purpose software, they are ill-suited for the kernel context. In particular, recent LLM-based techniques often treat bug reports and source code as plain text, lacking deep integration of kernel-specific knowledge, which limits their ability to identify root causes and achieve fine-grained localization. We present CoHiKer, a novel LLM-based FL technique tailored to the Linux kernel. CoHiKer introduces two key innovations: (1) contrastive reasoning, which identifies root causes by analyzing the behavioral divergence between carefully mutated passing and failing test cases, and (2) hierarchical context analysis, which systematically narrows the localization scope from files to methods by integrating crash reports, syscall semantics, inter-file dependencies, and kernel-specific features. Unlike prior techniques that rely on static understanding and full-code input, CoHiKer decomposes the localization task and enables structured LLM prompting to reason semantically over meaningful contexts. We evaluate CoHiKer on an extended Linux kernel bug dataset against five state-of-the-art baselines. CoHiKer consistently outperforms all competitors, improving Top-1 localization accuracy by up to 26.07% at the file level and 56.85% at the method level over state-of-the-art LLM-based baselines, while achieving up to 8.84× and 28.9× reductions in token consumption, respectively. Furthermore, CoHiKer demonstrates strong generalizability on the non-kernel dataset, with comparable gains (15.5% and 5.3% in Top-1 at file and method levels). These results demonstrate the promise of CoHiKer as a practical and accurate solution for kernel FL.
- 12 April 2026
- Published by Association for Computing Machinery (ACM)
- p. 717-729
Open AccessTowards Better Linux Kernel Fault Localization: Leveraging Contrastive Reasoning and Hierarchical Context Analysis
Debugging the Linux kernel remains a formidable challenge due to its vast codebase, complex architecture, and low-level programming intricacies. Effective fault localization (FL) is thus essential for efficient kernel debugging and maintenance. While existing FL techniques (both traditional and LLM-based) have shown promise in general-purpose software, they are ill-suited for the kernel context. In particular, recent LLM-based techniques often treat bug reports and source code as plain text, lacking deep integration of kernel-specific knowledge, which limits their ability to identify root causes and achieve fine-grained localization. We present CoHiKer, a novel LLM-based FL technique tailored to the Linux kernel. CoHiKer introduces two key innovations: (1) contrastive reasoning, which identifies root causes by analyzing the behavioral divergence between carefully mutated passing and failing test cases, and (2) hierarchical context analysis, which systematically narrows the localization scope from files to methods by integrating crash reports, syscall semantics, inter-file dependencies, and kernel-specific features. Unlike prior techniques that rely on static understanding and full-code input, CoHiKer decomposes the localization task and enables structured LLM prompting to reason semantically over meaningful contexts. We evaluate CoHiKer on an extended Linux kernel bug dataset against five state-of-the-art baselines. CoHiKer consistently outperforms all competitors, improving Top-1 localization accuracy by up to 26.07% at the file level and 56.85% at the method level over state-of-the-art LLM-based baselines, while achieving up to 8.84× and 28.9× reductions in token consumption, respectively. Furthermore, CoHiKer demonstrates strong generalizability on the non-kernel dataset, with comparable gains (15.5% and 5.3% in Top-1 at file and method levels). These results demonstrate the promise of CoHiKer as a practical and accurate solution for kernel FL.
- Open Access
- 12 April 2026
- Published by Association for Computing Machinery (ACM)
- p. 139-143
Jailbreaking Large Language Models via Multi-Task Embedding-based Prompt
With the widespread application of large language models (LLMs), jailbreak attacks that bypass their security mechanisms have become an important threat. Existing methods suffer from limited effectiveness, as they use only one attack pattern at a time, along with excessive query costs. To address these limitations, this paper proposes the Multi-Task Embedding-based Attack (MTEA), a novel jailbreak approach that embeds malicious instructions within three different tasks (e.g., Code Understanding, Language Translation, and Pattern Adherence). By exploiting LLMs’ reduced performance when handling multiple tasks concurrently, MTEA disrupts their safety alignment mechanisms. Extensive experiments on six widely used LLMs (including GPT-4o and Gemini-2.5-pro) using the AdvBench benchmark show that MTEA achieves 100% attack success rate (ASR) and 100% following rate (FR) across all models. It outperforms state-of-the-art baselines by 48.0 ∼ 49.7% in ASR and 60.4 ∼ 60.7% in FR. Even against advanced defense mechanisms (Perplexity Filter and SmoothLLM), MTEA maintains 100% ASR, while reducing query costs by 90% compared to baselines. These results demonstrate the effectiveness and efficiency of MTEA.
- 12 April 2026
- Published by Association for Computing Machinery (ACM)
- p. 139-143
Open AccessJailbreaking Large Language Models via Multi-Task Embedding-based Prompt
With the widespread application of large language models (LLMs), jailbreak attacks that bypass their security mechanisms have become an important threat. Existing methods suffer from limited effectiveness, as they use only one attack pattern at a time, along with excessive query costs. To address these limitations, this paper proposes the Multi-Task Embedding-based Attack (MTEA), a novel jailbreak approach that embeds malicious instructions within three different tasks (e.g., Code Understanding, Language Translation, and Pattern Adherence). By exploiting LLMs’ reduced performance when handling multiple tasks concurrently, MTEA disrupts their safety alignment mechanisms. Extensive experiments on six widely used LLMs (including GPT-4o and Gemini-2.5-pro) using the AdvBench benchmark show that MTEA achieves 100% attack success rate (ASR) and 100% following rate (FR) across all models. It outperforms state-of-the-art baselines by 48.0 ∼ 49.7% in ASR and 60.4 ∼ 60.7% in FR. Even against advanced defense mechanisms (Perplexity Filter and SmoothLLM), MTEA maintains 100% ASR, while reducing query costs by 90% compared to baselines. These results demonstrate the effectiveness and efficiency of MTEA.
- Open Access
LLM-based Vulnerability Detection at Project Scale: An Empirical Study
In this paper, we present the first comprehensive empirical study of specialized LLM-based detectors and compare them with traditional static analyzers at the project scale. Specifically, our study evaluates five latest and representative LLM-based methods and two traditional tools using: 1) an in-house benchmark of 222 known real-world vulnerabilities (C/C++ and Java) to assess detection capability, and 2) 24 active open-source projects, where we manually inspected 385 warnings to assess their practical usability and underlying root causes of failures. Our evaluation yields three key findings: First, while LLM-based detectors exhibit low recall on the in-house benchmark, they still uncover more unique vulnerabilities than traditional tools. Second, in open-source projects, both LLM-based and traditional tools generate substantial warnings but suffer from very high false discovery rates, hindering practical use. Our manual analysis further reveals shallow interprocedural reasoning and misidentified source/sink pairs as primary failure causes, with LLM-based tools exhibiting additional unique failures. Finally, LLM-based methods incurs substantial computational costs-hundreds of thousands to hundreds of millions of tokens and multi-hour to multi-day runtimes. Overall, our findings underscore critical limitations in the robustness, reliability, and scalability of current LLM-based detectors. We ultimately summarize a set of implications for future research toward more effective and practical project-scale vulnerability detection.
Open AccessLLM-based Vulnerability Detection at Project Scale: An Empirical Study
In this paper, we present the first comprehensive empirical study of specialized LLM-based detectors and compare them with traditional static analyzers at the project scale. Specifically, our study evaluates five latest and representative LLM-based methods and two traditional tools using: 1) an in-house benchmark of 222 known real-world vulnerabilities (C/C++ and Java) to assess detection capability, and 2) 24 active open-source projects, where we manually inspected 385 warnings to assess their practical usability and underlying root causes of failures. Our evaluation yields three key findings: First, while LLM-based detectors exhibit low recall on the in-house benchmark, they still uncover more unique vulnerabilities than traditional tools. Second, in open-source projects, both LLM-based and traditional tools generate substantial warnings but suffer from very high false discovery rates, hindering practical use. Our manual analysis further reveals shallow interprocedural reasoning and misidentified source/sink pairs as primary failure causes, with LLM-based tools exhibiting additional unique failures. Finally, LLM-based methods incurs substantial computational costs-hundreds of thousands to hundreds of millions of tokens and multi-hour to multi-day runtimes. Overall, our findings underscore critical limitations in the robustness, reliability, and scalability of current LLM-based detectors. We ultimately summarize a set of implications for future research toward more effective and practical project-scale vulnerability detection.
- Open Access
- 9 October 2025
- Published by Association for Computing Machinery (ACM) in Proceedings of the ACM on Programming Languages
- Vol. 9  (OOPSLA2) , 3093-3119
DESIL: Detecting Silent Bugs in MLIR Compiler Infrastructure
MLIR (Multi-Level Intermediate Representation) compiler infrastructure provides an efficient framework for introducing a new abstraction level for programming languages and domain-specific languages. It has attracted widespread attention in recent years and has been applied in various domains, such as deep learning compiler construction. Recently, several MLIR compiler fuzzing techniques, such as MLIRSmith and MLIRod, have been proposed. However, none of them can detect silent bugs, i.e., bugs that incorrectly optimize code silently. The difficulty in detecting silent bugs arises from two main aspects: (1) UB-Free Program Generation: Generates programs that are free from undefined behaviors to suit the non-UB assumptions required by compiler optimizations. (2) Lowering Support: Converts the given MLIR program into an executable form with a suitable lowering path that reduces redundant lowering passes and improves the efficiency of fuzzing. To address the above issues, we propose DESIL. DESIL enables silent bug detection by defining a set of UB-elimination rules based on the MLIR documentation and applying them to input programs. To convert dialects in the MLIR program into executable form, DESIL designs a lowering path optimization strategy to convert the dialects in the given MLIR program into executable form. Furthermore, DESIL incorporates the differential testing for silent bug detection. It introduces an operation-aware optimization recommendation strategy into the compilation process to generate diverse executable files. We applied DESIL to the latest revisions of the MLIR compiler infrastructure. It detected 23 silent bugs and 19 crash bugs, of which 17/16 have been confirmed or fixed.
- 9 October 2025
- Published by Association for Computing Machinery (ACM) in Proceedings of the ACM on Programming Languages
- Vol. 9  (OOPSLA2) , 3093-3119
Open AccessDESIL: Detecting Silent Bugs in MLIR Compiler Infrastructure
MLIR (Multi-Level Intermediate Representation) compiler infrastructure provides an efficient framework for introducing a new abstraction level for programming languages and domain-specific languages. It has attracted widespread attention in recent years and has been applied in various domains, such as deep learning compiler construction. Recently, several MLIR compiler fuzzing techniques, such as MLIRSmith and MLIRod, have been proposed. However, none of them can detect silent bugs, i.e., bugs that incorrectly optimize code silently. The difficulty in detecting silent bugs arises from two main aspects: (1) UB-Free Program Generation: Generates programs that are free from undefined behaviors to suit the non-UB assumptions required by compiler optimizations. (2) Lowering Support: Converts the given MLIR program into an executable form with a suitable lowering path that reduces redundant lowering passes and improves the efficiency of fuzzing. To address the above issues, we propose DESIL. DESIL enables silent bug detection by defining a set of UB-elimination rules based on the MLIR documentation and applying them to input programs. To convert dialects in the MLIR program into executable form, DESIL designs a lowering path optimization strategy to convert the dialects in the given MLIR program into executable form. Furthermore, DESIL incorporates the differential testing for silent bug detection. It introduces an operation-aware optimization recommendation strategy into the compilation process to generate diverse executable files. We applied DESIL to the latest revisions of the MLIR compiler infrastructure. It detected 23 silent bugs and 19 crash bugs, of which 17/16 have been confirmed or fixed.
- Top Cited PapersOpen Access
- 17 August 2025
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Software Engineering and Methodology
- Vol. 34  (7) , 1-28
Hybrid Automated Program Repair by Combining Large Language Models and Program Analysis
Automated Program Repair (APR) has garnered significant attention due to its potential to streamline the bug repair process for human developers. Recently, LLM-based APR methods have shown promise in repairing real-world bugs. However, existing APR methods often utilize patches generated by LLMs without further optimization, resulting in reduced effectiveness due to the lack of program-specific knowledge. Furthermore, the evaluations of these APR methods have typically been conducted under the assumption of perfect fault localization, which may not accurately reflect their real-world effectiveness. To address these limitations, this article introduces an innovative APR approach called G iant R epair . Our approach leverages the insight that LLM-generated patches, although not necessarily correct, offer valuable guidance for the patch generation process. Based on this insight, G iant R epair first constructs patch skeletons from LLM-generated patches to confine the patch space, and then generates high-quality patches tailored to specific programs through context-aware patch generation by instantiating the skeletons. To evaluate the performance of our approach, we conduct two large-scale experiments. The results demonstrate that G iant R epair not only effectively repairs more bugs (an average of 27.78% on Defects4J v1.2 and 23.40% on Defects4J v2.0) than using LLM-generated patches directly, but also outperforms state-of-the-art APR methods by repairing at least 42 and 7 more bugs under perfect and automated fault localization scenarios, respectively.
- 17 August 2025
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Software Engineering and Methodology
- Vol. 34  (7) , 1-28
Top Cited PapersOpen AccessHybrid Automated Program Repair by Combining Large Language Models and Program Analysis
Automated Program Repair (APR) has garnered significant attention due to its potential to streamline the bug repair process for human developers. Recently, LLM-based APR methods have shown promise in repairing real-world bugs. However, existing APR methods often utilize patches generated by LLMs without further optimization, resulting in reduced effectiveness due to the lack of program-specific knowledge. Furthermore, the evaluations of these APR methods have typically been conducted under the assumption of perfect fault localization, which may not accurately reflect their real-world effectiveness. To address these limitations, this article introduces an innovative APR approach called G iant R epair . Our approach leverages the insight that LLM-generated patches, although not necessarily correct, offer valuable guidance for the patch generation process. Based on this insight, G iant R epair first constructs patch skeletons from LLM-generated patches to confine the patch space, and then generates high-quality patches tailored to specific programs through context-aware patch generation by instantiating the skeletons. To evaluate the performance of our approach, we conduct two large-scale experiments. The results demonstrate that G iant R epair not only effectively repairs more bugs (an average of 27.78% on Defects4J v1.2 and 23.40% on Defects4J v2.0) than using LLM-generated patches directly, but also outperforms state-of-the-art APR methods by repairing at least 42 and 7 more bugs under perfect and automated fault localization scenarios, respectively.
- Open Access
Improving Compiler Bug Isolation by Leveraging Large Language Models
Compilers play a foundational role in building reliable software systems, and bugs within them can lead to catastrophic consequences. The compilation process typically involves hundreds of files, making traditional automated bug isolation techniques inapplicable due to scalability or effectiveness issues. Current mainstream compiler bug localization techniques have limitations in test program mutation and resource consumption. Inspired by the recent advances of pre-trained Large Language Models (LLMs), we propose an innovative approach named AutoCBI, which (1) uses LLMs to summarize compiler file functions and (2) employs specialized prompts to guide LLM in reordering suspicious file rankings. This approach leverages four types of information: the failing test program, source file function summaries, lists of suspicious files identified through analyzing test coverage, as well as compilation configurations with related output messages, resulting in a refined ranking of suspicious files. Our evaluation of AutoCBI against state-of-the-art approaches (DiWi, RecBi and FuseFL) on 120 real-world bugs from the widely-used GCC and LLVM compilers demonstrates its effectiveness. Specifically, AutoCBI isolates 66.67%/69.23%, 300%/340%, and 100%/57.14% more bugs than RecBi, DiWi, and FuseFL, respectively, in the Top-1 ranked results for GCC/LLVM. Additionally, the ablation study underscores the significance of each component in our approach.
Open AccessImproving Compiler Bug Isolation by Leveraging Large Language Models
Compilers play a foundational role in building reliable software systems, and bugs within them can lead to catastrophic consequences. The compilation process typically involves hundreds of files, making traditional automated bug isolation techniques inapplicable due to scalability or effectiveness issues. Current mainstream compiler bug localization techniques have limitations in test program mutation and resource consumption. Inspired by the recent advances of pre-trained Large Language Models (LLMs), we propose an innovative approach named AutoCBI, which (1) uses LLMs to summarize compiler file functions and (2) employs specialized prompts to guide LLM in reordering suspicious file rankings. This approach leverages four types of information: the failing test program, source file function summaries, lists of suspicious files identified through analyzing test coverage, as well as compilation configurations with related output messages, resulting in a refined ranking of suspicious files. Our evaluation of AutoCBI against state-of-the-art approaches (DiWi, RecBi and FuseFL) on 120 real-world bugs from the widely-used GCC and LLVM compilers demonstrates its effectiveness. Specifically, AutoCBI isolates 66.67%/69.23%, 300%/340%, and 100%/57.14% more bugs than RecBi, DiWi, and FuseFL, respectively, in the Top-1 ranked results for GCC/LLVM. Additionally, the ablation study underscores the significance of each component in our approach.
- Open Access
Empirical Evaluation of Large Language Models in Automated Program Repair
The increasing prevalence of software bugs has made automated program repair (APR) a key research focus. Large language models (LLMs) offer new opportunities for APR, but existing studies mostly rely on smaller, earlier-generation models and Java benchmarks. The repair capabilities of modern, large-scale LLMs across diverse languages and scenarios remain underexplored. To address this, we conduct a comprehensive empirical study of four open-source LLMs, CodeLlama, LLaMA, StarCoder, and DeepSeek-Coder, spanning 7B to 33B parameters, diverse architectures, and purposes. We evaluate them across two bug scenarios (enterprise-grades and algorithmic), three languages (Java, C/C++, Python), and four prompting strategies, analyzing over 600K generated patches on six benchmarks. Key findings include: (1) model specialization (e.g., CodeLlama) can outperform larger general-purpose models (e.g., LLaMA); (2) repair performance does not scale linearly with model size; (3) correct patches often appear early in generation; and (4) prompts significantly affect results. These insights offer practical guidance for designing effective and efficient LLM-based APR systems.
Open AccessEmpirical Evaluation of Large Language Models in Automated Program Repair
The increasing prevalence of software bugs has made automated program repair (APR) a key research focus. Large language models (LLMs) offer new opportunities for APR, but existing studies mostly rely on smaller, earlier-generation models and Java benchmarks. The repair capabilities of modern, large-scale LLMs across diverse languages and scenarios remain underexplored. To address this, we conduct a comprehensive empirical study of four open-source LLMs, CodeLlama, LLaMA, StarCoder, and DeepSeek-Coder, spanning 7B to 33B parameters, diverse architectures, and purposes. We evaluate them across two bug scenarios (enterprise-grades and algorithmic), three languages (Java, C/C++, Python), and four prompting strategies, analyzing over 600K generated patches on six benchmarks. Key findings include: (1) model specialization (e.g., CodeLlama) can outperform larger general-purpose models (e.g., LLaMA); (2) repair performance does not scale linearly with model size; (3) correct patches often appear early in generation; and (4) prompts significantly affect results. These insights offer practical guidance for designing effective and efficient LLM-based APR systems.
- 18 March 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Software Engineering
- Vol. 51  (5) , 1399-1414
Evaluating Spectrum-Based Fault Localization on Deep Learning Libraries
Deep learning (DL) libraries have become increasingly popular and their quality assurance is also gaining significant attention. Although many fault detection techniques have been proposed, effective fault localization techniques tailored to DL libraries are scarce. Due to the unique characteristics of DL libraries (e.g., complicated code architecture supporting DL model training and inference with extensive multidimensional tensor calculations), the effectiveness of existing fault localization techniques for traditional software is also unknown on DL library faults. To bridge this gap, we conducted the first empirical study to investigate the effectiveness of fault localization on DL libraries. Specifically, we evaluated spectrum-based fault localization (SBFL) due to its high generalizability and affordable overhead on such complicated libraries. Based on the key aspects in SBFL, our study investigated the effectiveness of SBFL with different sources of passing test cases (including human-written, fuzzer-generated, and mutation-based test cases) and various suspicious value calculation methods. In particular, mutation-based test cases are produced by our designed rule-based mutation technique and LLM-based mutation technique tailored to DL library faults. To enable our extensive study, we built the first benchmark (Defects4DLL), which contains 120 real-world faults in PyTorch and TensorFlow with easy-to-use experimental environments. Our study delivered a series of useful findings. For example, the rule-based approach is effective in localizing crash faults in DL libraries, successfully localizing 44.44% of crash faults within Top-10 functions and 74.07% of crash faults within Top-10 files, while the passing test cases from DL library fuzzers perform poorly on this task. Furthermore, based on our findings on the complementarity of different sources, we designed a hybrid technique by effectively integrating human-written, LLM-mutated, rule-based mutated test cases, which further achieves 31.48%~61.36% improvements over each single source in terms of the number of detected faults within Top-5 files.
- 18 March 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Software Engineering
- Vol. 51  (5) , 1399-1414
Evaluating Spectrum-Based Fault Localization on Deep Learning Libraries
Deep learning (DL) libraries have become increasingly popular and their quality assurance is also gaining significant attention. Although many fault detection techniques have been proposed, effective fault localization techniques tailored to DL libraries are scarce. Due to the unique characteristics of DL libraries (e.g., complicated code architecture supporting DL model training and inference with extensive multidimensional tensor calculations), the effectiveness of existing fault localization techniques for traditional software is also unknown on DL library faults. To bridge this gap, we conducted the first empirical study to investigate the effectiveness of fault localization on DL libraries. Specifically, we evaluated spectrum-based fault localization (SBFL) due to its high generalizability and affordable overhead on such complicated libraries. Based on the key aspects in SBFL, our study investigated the effectiveness of SBFL with different sources of passing test cases (including human-written, fuzzer-generated, and mutation-based test cases) and various suspicious value calculation methods. In particular, mutation-based test cases are produced by our designed rule-based mutation technique and LLM-based mutation technique tailored to DL library faults. To enable our extensive study, we built the first benchmark (Defects4DLL), which contains 120 real-world faults in PyTorch and TensorFlow with easy-to-use experimental environments. Our study delivered a series of useful findings. For example, the rule-based approach is effective in localizing crash faults in DL libraries, successfully localizing 44.44% of crash faults within Top-10 functions and 74.07% of crash faults within Top-10 files, while the passing test cases from DL library fuzzers perform poorly on this task. Furthermore, based on our findings on the complementarity of different sources, we designed a hybrid technique by effectively integrating human-written, LLM-mutated, rule-based mutated test cases, which further achieves 31.48%~61.36% improvements over each single source in terms of the number of detected faults within Top-5 files.