Chain of Targeted Verification Questions to Improve the Reliability of Code Generated by LLMs
Open Access
- 10 July 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 122-130
Abstract
LLM-based assistants, such as GitHub Copilot and ChatGPT, have the potential to generate code that fulfills a programming task described in a natural language description, referred to as a prompt. The widespread accessibility of these assistants enables users with diverse backgrounds to generate code and integrate it into software projects. However, studies show that code generated by LLMs is prone to bugs and may miss various corner cases in task specifications. Presenting such buggy code to users can impact their reliability and trust in LLM-based assistants. Moreover, significant efforts are required by the user to detect and repair any bug present in the code, especially if no test cases are available. In this study, we propose a self-refinement method aimed at improving the reliability of code generated by LLMs by minimizing the number of bugs before execution, without human intervention, and in the absence of test cases. Our approach is based on targeted Verification Questions (VQs) to identify potential bugs within the initial code. These VQs target various nodes within the Abstract Syntax Tree (AST) of the initial code, which have the potential to trigger specific types of bug patterns commonly found in LLM-generated code. Finally, our method attempts to repair these potential bugs by re-prompting the LLM with the targeted VQs and the initial code. Our evaluation, based on programming tasks in the CoderEval dataset, demonstrates that our proposed method outperforms state-of-the-art methods by decreasing the number of targeted errors in the code between 21% to 62% and improving the number of executable code instances to 13%.Keywords
All Related Versions
Funding Information
- Natural Sciences and Engineering Research Council of Canada (NSERC) ()
This publication has 23 references indexed in Scilit:
- Better Patching Using LLM Prompting, via Self-Consistency2023
- GitHub Copilot AI pair programmer: Asset or Liability?Journal of Systems and Software, 2023
- On the Robustness of Code Generation Techniques: An Empirical Study on GitHub Copilot2023
- Automated Repair of Programs from Large Language Models2023
- Automated Program Repair in the Era of Large Pre-trained Language Models2023
- A Brief Overview of ChatGPT: The History, Status Quo and Potential Future DevelopmentIEEE/CAA Journal of Automatica Sinica, 2023
- Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models2022
- Understanding Code Smell Detection via Code Review: A Study of the OpenStack Community2021
- I think I made a mistake: How do prospective teachers elicit the thinking of a student who has made a mistake?Journal of Mathematics Teacher Education, 2020
- Confusion in Code Reviews: Reasons, Impacts, and Coping Strategies2019