Explainable automated debugging via large language model-driven scientific debugging
Top Cited Papers
Open Access
- 18 December 2024
- journal article
- research article
- Published by Springer Nature in Empirical Software Engineering
- Vol. 30 (2) , 1-28
Abstract
Automated debugging techniques have the potential to reduce developer effort in debugging. However, while developers want rationales for the provided automatic debugging results, existing techniques are ill-suited to provide them, as their deduction process differs significantly froof human developers. Inspired by the way developers interact with code when debugging, we propose Automated Scientific Debugging (AutoSD), a technique that prompts large language models to automatically generate hypotheses, uses debuggers to interact with buggy code, and thus automatically reach conclusions prior to patch generation. In doing so, we aim to produce explanations of how a specific patch has been generated, with the hope that these explanations will lead to enhanced developer decision-making. Our empirical analysis on three program repair benchmarks shows that AutoSDperforms competitively with other program repair baselines, and that it can indicate when it is confident in its results. Furthermore, we perform a human study with 20 participants to evaluate AutoSD-generated explanations. Participants with access to explanations judged patch correctness more accurately in five out of six real-world bugs studied. Furthermore, 70% of participants answered that they wanted explanations when using repair tools, and 55% answered that they were satisfied with the Scientific Debugging presentation.Keywords
All Related Versions
Funding Information
- Institute of Information & Communications Technology Planning & Evaluation (IITP2022-0-00995, IITP2022-0-00995)
- National Research Foundation of Korea (RS-2023-00208998, RS-2023-00208998)
- Institute of Information & Communications Technology Planning & Evaluation (RS-2022-00155958)
This publication has 47 references indexed in Scilit:
- Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AIInformation Fusion, 2020
- Explainable Software Bot Contributions: Case Study of Automated Bug Fixes2019
- SapFix: Automated End-to-End Repair at Scale2019
- Astor: Exploring the design space of generate-and-validate program repair beyond GenProgJournal of Systems and Software, 2019
- Automatically generating commit messages from diffs using neural machine translation2017
- Precise Condition Synthesis for Program Repair2017
- How Developers Debug Software — The DBGBENCH Dataset2017
- Debugging Revisited: Toward Understanding the Debugging Needs of Contemporary Software Developers2013
- A practical guide to controlled experiments of software engineering tools with human participantsEmpirical Software Engineering, 2013
- Some psychological evidence on how people debug computer programsInternational Journal of Man-Machine Studies, 1975