What It Would Take to Use Mutation Testing in Industry—A Study at Facebook
Authors:
- 1 May 2021
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 268-277
Abstract
Traditionally, mutation testing generates an abun-dance of small deviations of a program, called mutants. At industrial systems the scale and size of Facebook’s, doing this is infeasible. We should not create mutants that the test suite would likely fail on or that give no actionable signal to developers. To tackle this problem, in this paper, we semi-automatically learn error-inducing patterns from a corpus of common Java coding errors and from changes that caused operational anomalies at Facebook specifically. We combine the mutations with instrumentation that measures which tests exactly visited the mutated piece of code. Results on more than 15,000 generated mutants show that more than half of the generated mutants survive Facebook’s rigorous test suite of unit, integration, and system tests. Moreover, in a case study with 26 developers, all but two expressed that the mutation exposed a lack of testing in principle. As such, almost half of the 26 would actually act on the mutant presented to them by adapting an existing or creating a new test. The others did not for a variety of reasons often outside the scope of mutation testing. It remains a practical challenge how we can include such external information to increase the actionability rate on mutants.Keywords
All Related Versions
This publication has 28 references indexed in Scilit:
- Are mutants a valid substitute for real faults in software testing?2014
- Defects4J: a database of existing faults to enable controlled testing studies for Java programs2014
- Coverage is not strongly correlated with test suite effectiveness2014
- Code coverage for suite evaluation by developers2014
- Towards Incremental Mutation TestingElectronic Notes in Theoretical Computer Science, 2013
- On guiding the augmentation of an automated test suite via mutation analysisEmpirical Software Engineering, 2008
- Empirical evaluation of the textual differencing regression testing technique2002
- Toward the determination of sufficient mutant operators for C†Software Testing, Verification and Reliability, 2001
- Mutation 2000: Uniting the Orthogonal2001
- All-uses vs mutation testing: An experimental comparison of effectivenessJournal of Systems and Software, 1997