Can OpenAI's codex fix bugs?
Authors:
Open Access
- 19 May 2022
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 69-75
Abstract
OpenAI's Codex, a GPT-3 like model trained on a large code corpus, has made headlines in and outside of academia. Given a short user-provided description, it is capable of synthesizing code snippets that are syntactically and semantically valid in most cases. In this work, we want to investigate whether Codex is able to localize and fix bugs, two important tasks in automated program repair. Our initial evaluation uses the multi-language QuixBugs benchmark (40 bugs in both Python and Java). We find that, despite not being trained for APR, Codex is surprisingly effective, and competitive with recent state of the art techniques. Our results also show that Codex is more successful at repairing Python than Java, fixing 50% more bugs in Python.Keywords
This publication has 15 references indexed in Scilit:
- BugsJS: a Benchmark of JavaScript Bugs2019
- LSRepair: Live Search of Fix Ingredients for Automated Program Repair2018
- ARJA: Automated Repair of Java Programs via Multi-Objective Genetic ProgrammingIEEE Transactions on Software Engineering, 2018
- QuixBugs: a multi-lingual program repair benchmark set based on the quixey challenge2017
- Elixir: Effective object-oriented program repair2017
- Angelix2016
- Nopol: Automatic Repair of Conditional Statement Bugs in Java ProgramsIEEE Transactions on Software Engineering, 2016
- Defects4J: a database of existing faults to enable controlled testing studies for Java programs2014
- GenProg: A Generic Method for Automatic Software RepairIEEE Transactions on Software Engineering, 2011
- Maintaining mental models2006