Use and Misuse of the Term “Experiment” in Mining Software Repositories Research
- 24 September 2021
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Software Engineering
- Vol. 48 (11) , 4229-4248
Abstract
The significant momentum and importance of Mining Software Repositories (MSR) in Software Engineering (SE) has fostered new opportunities and challenges for extensive empirical research. However, MSR researchers seem to struggle to characterize the empirical methods they use into the existing empirical SE body of knowledge. This is especially the case of MSR experiments. To provide evidence on the special characteristics of MSR experiments and their differences with experiments traditionally acknowledged in SE so far, we elicited the hallmarks that differentiate an experiment from other types of empirical studies and characterized the hallmarks and types of experiments in MSR. We analyzed MSR literature obtained from a small-scale systematic mapping study to assess the use of the term experiment in MSR. We found that 19% of the papers claiming to be an experiment are indeed not an experiment at all but also observational studies, so they use the term in a misleading way. From the remaining 81% of the papers, only one of them refers to a genuine controlled experiment while the others stand for experiments with limited control. MSR researchers tend to overlook such limitations, compromising the interpretation of the results of their studies. We provide recommendations and insights to support the improvement of MSR experiments.Keywords
All Related Versions
Funding Information
- Spanish project (MCI PID2020-117191RB-I00, AEI/10.13039/501100011033)
This publication has 52 references indexed in Scilit:
- When Natural Experiments Are Neither Natural nor ExperimentsAmerican Political Science Review, 2012
- A Survey on Mining Software RepositoriesIEICE Transactions on Information and Systems, 2012
- An overview of randomization techniques: An unbiased assessment of outcome in clinical researchJournal of Human Reproductive Sciences, 2011
- A systematic review of quasi-experiments in software engineeringInformation and Software Technology, 2009
- Alternatives to the Randomized Controlled TrialAmerican Journal of Public Health, 2008
- A survey and taxonomy of approaches for mining software repositories in the context of software evolutionJournal of Software Maintenance and Evolution: Research and Practice, 2007
- A Comparison of Experimental and Observational Data Analyses2004
- Preliminary guidelines for empirical research in software engineeringIEEE Transactions on Software Engineering, 2002
- Experimental Validity: Brunswik, Campbell, Cronbach, and Enduring IssuesReview of General Psychology, 2000
- Random versus nonrandom assignment in controlled experiments: Do you get the same answer?Journal of Consulting and Clinical Psychology, 1996