Challenging Bug Prediction and Repair Models with Synthetic Bugs
Authors:
- 8 September 2025
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 133-144
Abstract
Bugs are essential in software engineering; many research studies in the past decades have been proposed to detect, localize, and repair bugs in software systems. Effectiveness evaluation of such techniques requires complex bugs, i.e., those that are hard to detect through testing and hard to repair through debugging. From the classic software engineering point of view, a hard-to-repair bug differs from the correct code in multiple locations, making it hard to localize and repair. Hard-to-detect bugs, on the other hand, manifest themselves under specific test inputs and reachability conditions. These two objectives, i.e., generating hard-to-detect and hard-to-repair bugs, are mostly aligned; a bug generation technique can change multiple statements to be covered only under a specific set of inputs. However, these two objectives conflict in the learning-based techniques: A bug should have a similar code representation to the correct code in the training data to challenge a bug prediction model to distinguish them. The hard-to-repair bug definition remains the same but with a caveat: the more a bug differs from the original code (at multiple locations), the more distant their representations are and easier to detect. This demands new techniques to generate bugs to complement existing bug datasets to challenge learning-based bug prediction and repair techniques.(p)(/p)We propose BugFarm to transform arbitrary code into multiple hard-to-detect and hard-to-repair bugs. BugFarm mutates code in multiple locations (hard-to-repair) but leverages attention analysis to only change the least attended locations by the underlying model (hard-to-detect). Our comprehensive evaluation of 435k+ bugs from over 1.9M mutants generated by BugFarm and two alternative approaches demonstrates our superiority in generating bugs that are hard to detect by learning-based bug prediction approaches (up to 40.53% higher False Negative Rate and 10.76%, 5.2%, 28.93%, and 20.53% lower Accuracy, Precision, Recall, and F1 score) and hard to repair by state-of-the-art learning-based program repair technique (28% repair success rate compared to 36% and 49% of LEAM and μBERT bugs). BugFarm is efficient, i.e., it takes nine seconds to mutate a code with no training overhead.Keywords
All Related Versions
This publication has 30 references indexed in Scilit:
- The Plastic Surgery Hypothesis in the Era of Large Language Models2023
- VULGEN: Realistic Vulnerability Generation Via Pattern Mining and Deep Learning2023
- What It Would Take to Use Mutation Testing in Industry—A Study at Facebook2021
- Hazards of data leakage in machine learning: a study on classification of breast cancer using deep neural networks2020
- Learning How to Mutate Source Code from Bug-Fixes2019
- BugSwarm: Mining and Continuously Growing a Dataset of Reproducible Failures and Fixes2019
- Deep Learning for Just-in-Time Defect Prediction2015
- Defects4J: a database of existing faults to enable controlled testing studies for Java programs2014
- The major mutation framework: efficient and scalable mutation analysis for Java2014
- Higher Order Mutation TestingInformation and Software Technology, 2009