A new perspective on the competent programmer hypothesis through the reproduction of real faults with repeated mutations
Authors:
Open Access
- 29 February 2024
- journal article
- research article
- Published by Wiley in Software Testing, Verification and Reliability
- Vol. 34 (3)
Abstract
The competent programmer hypothesis is one of the fundamental assumptions of mutation testing, which claims that most programmers are competent enough to create correct or almost correct source code. This implies that faults should usually manifest through small variations of the correct code. Consequently, researchers assumed that the synthetic faults injected in source code through the mutation operators closely resemble the real faults. Unfortunately, it is still unclear whether the competent programmer hypothesis holds, as past research presents contradictory claims. Within this article, we provide a new perspective on the competent programmer hypothesis and its relation to mutation testing. We try to re‐create real‐world faults through chains of mutations to understand if there is a direct link between mutation testing and faults. The lengths of these paths help us to understand if the source code is really almost correct, or if large variations are required. Our experiments used a state‐of‐the‐art benchmark database of real faults named Defects4J 2.0.0. It contains 835 reproducible real‐world faults in 17 open‐source projects that comprise a total of 1044 bug‐fix pairs of files. Our results indicate that while the competent programmer hypothesis seems to be true, mutation testing is missing important operators to generate representative real‐world faults.Keywords
All Related Versions
This publication has 21 references indexed in Scilit:
- A fine-grained data set and analysis of tangling in bug fixing commitsEmpirical Software Engineering, 2022
- Problems with SZZ and features: An empirical study of the state of practice of defect prediction data collectionEmpirical Software Engineering, 2022
- What It Would Take to Use Mutation Testing in Industry—A Study at Facebook2021
- The impact of tangled code changes on defect prediction modelsEmpirical Software Engineering, 2015
- Mutations: How Close are they to Real Faults?2014
- It's not a bug, it's a feature: How misclassification impacts bug prediction2013
- Experimentation in Software Engineering2012
- Recognizing authors: an examination of the consistent programmer hypothesisSoftware Testing, Verification and Reliability, 2010
- MuJava: an automated class mutation systemSoftware Testing, Verification and Reliability, 2004
- Hints on Test Data Selection: Help for the Practicing ProgrammerComputer, 1978