Better Data Labelling with EMBLEM (and how that Impacts Defect Prediction)
Preprint
Abstract
Standard automatic methods for recognizing problematic development commits can be greatly improved via the incremental application of human+artificial expertise. In this approach, called EMBLEM, an AI tool first explore the software development process to label commits that are most problematic. Humans then apply their expertise to check those labels (perhaps resulting in the AI updating the support vectors within their SVM learner). We recommend this human+AI partnership, for several reasons. When a new domain is encountered, EMBLEM can learn better ways to label which comments refer to real problems. Also, in studies with 9 open source software projects, labelling via EMBLEM's incremental application of human+AI is at least an order of magnitude cheaper than existing methods ($\approx$ eight times). Further, EMBLEM is very effective. For the data sets explored here, EMBLEM better labelling methods significantly improved $P_{opt}20$ and G-scores performance in nearly all the projects studied here.
Keywords
All Related Versions
This publication has 78 references indexed in Scilit:
- FAST2: An intelligent assistant for finding relevant papersExpert Systems with Applications, 2019
- Finding better active learners for faster literature reviewsEmpirical Software Engineering, 2018
- Curating GitHub for engineered software projectsEmpirical Software Engineering, 2017
- Reaction Mechanism Generator: Automatic construction of chemical kinetic mechanismsComputer Physics Communications, 2016
- An overview of the Amber biomolecular simulation packageWIREs Computational Molecular Science, 2012
- Can traditional fault prediction models be used for vulnerability prediction?Empirical Software Engineering, 2011
- Data Mining: Practical Machine Learning Tools and Techniques2011
- Defect prediction from static code features: current results, limitations, new approachesAutomated Software Engineering, 2010
- libMesh : a C++ library for parallel adaptive mesh refinement/coarsening simulationsEngineering with Computers, 2006
- Fast Parallel Algorithms for Short-Range Molecular DynamicsJournal of Computational Physics, 1995