Taxonomy of real faults in deep learning systems
Authors:
Open Access
- 27 June 2020
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1110-1121
Abstract
The growing application of deep neural networks in safety-critical domains makes the analysis of faults that occur in such systems of enormous importance. In this paper we introduce a large taxonomy of faults in deep learning (DL) systems. We have manually analysed 1059 artefacts gathered from GitHub commits and issues of projects that use the most popular DL frameworks (TensorFlow, Keras and PyTorch) and from related Stack Overflow posts. Structured interviews with 20 researchers and practitioners describing the problems they have encountered in their experience have enriched our taxonomy with a variety of additional faults that did not emerge from the other two sources. Our final taxonomy was validated with a survey involving an additional set of 21 developers, confirming that almost all fault categories (13/15) were experienced by at least 50% of the survey participants.Keywords
All Related Versions
Funding Information
- European Research Council (787703)
This publication has 15 references indexed in Scilit:
- An empirical study on TensorFlow program bugs2018
- An Empirical Study on Real Bugs for Machine Learning Programs2017
- Crowdsourced Knowledge on Stack Overflow2017
- Taxonomies in software engineering: A Systematic mapping study and a revised taxonomy development methodInformation and Software Technology, 2017
- Are mutants a valid substitute for real faults in software testing?2014
- An Empirical Study of Bugs in Machine Learning Systems2012
- Defect categorization2008
- Experiences from Conducting Semi-structured Interviews in Empirical Software Engineering Research2005
- Is mutation an appropriate tool for testing experiments?2005
- Qualitative methods in empirical studies of software engineeringIEEE Transactions on Software Engineering, 1999