An Empirical Study of Bugs in Machine Learning Systems
- 1 November 2012
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- No. 10719458,p. 271-280
Abstract
Many machine learning systems that include various data mining, information retrieval, and natural language processing code and libraries are used in real world applications. Search engines, internet advertising systems, product recommendation systems are sample users of these algorithm-intensive code and libraries. Machine learning code and toolkits have also been used in many recent studies on software mining and analytics that aim to automate various software engineering tasks. With the increasing number of important applications of machine learning systems, the reliability of such systems is also becoming increasingly important. A necessary step for ensuring reliability of such systems is to understand the features and characteristics of bugs occurred in the systems. A number of studies have investigated bugs and fixes in various software systems, but none focuses on machine learning systems. Machine learning systems are unique due to their algorithm-intensive nature and applications to potentially large-scale data, and thus deserve a special consideration. In this study, we fill the research gap by performing an empirical study on the bugs in machine learning systems. We analyze three systems, Apache Mahout, Lucene, and OpenNLP, which are data mining, information retrieval, and natural language processing tools respectively. We look into their bug databases and code repositories, analyze a sample set of bugs and corresponding fixes, and label the bugs into various categories. Our study finds that 22.6% of the bugs belong to the algorithm/method category, 15.6% of the bugs belong to the non-functional category, and 13% of the bugs belong to the assignment/initialization category. We also report the relationship between bug categories and bug severities, the time and effort needed to fix the bugs, and bug impacts. We highlight several bug categories that deserve attention in future research.Keywords
This publication has 42 references indexed in Scilit:
- A Comparative Study of the Performance of IR Models on Duplicate Bug Detection2012
- Towards more accurate retrieval of duplicate bug reports2011
- Impact of Installation Counts on Perceived Quality: A Case Study on Debian2011
- Comparing Mining Algorithms for Predicting the Severity of a Reported Bug2011
- A discriminative model approach for accurate duplicate bug report retrieval2010
- BugFix: A learning-based tool to assist developers in fixing bugs2009
- Defect categorization2008
- An approach to detecting duplicate bug reports using natural language and execution information2008
- Automated duplicate detection for bug tracking systems2008
- Detection of Duplicate Defect Reports Using Natural Language Processing29th International Conference on Software Engineering (ICSE'07), 2007