Do Not Trust Build Results at Face Value - An Empirical Study of 30 Million CPAN Builds
Authors:
- Mahdis ZolfaghariniaMZMahdis Zolfagharinia
- Bram AdamsBABram Adams
- Yann-Gael GuehenucYGYann-Gael Guehenuc
- 1 May 2017
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 312-322
Abstract
Continuous Integration (CI) is a cornerstone of modern quality assurance, providing on-demand builds (compilation and tests) of code changes or software releases. Despite the myriad of CI tools and frameworks, the basic activity of interpreting build results is not straightforward, due to not only the number of builds being performed but also, and especially, due to the phenomenon of build inflation, according to which one code change can be built on dozens of different operating systems, run-time environments and hardware architectures. As existing work mostly ignored this inflation, this paper performs a large-scale empirical study of the impact of OS and run-time environment on build failures on 30 million builds of the CPAN ecosystem's CI environment. We observe the evolution of build failures over time, and investigate the impact of OSes and environments on build failures. We show that distributions may fail differently on different OSes and environments and, thus, that the results of CI require careful filtering and selection to identify reliable failure data.Keywords
This publication has 23 references indexed in Scilit:
- Quality and productivity outcomes relating to continuous integration in GitHub2015
- The highways and country roads to continuous deploymentIEEE Software, 2015
- Tracing back the history of commits in low-tech reviewing environments2014
- Continuous Integration in a Social-Coding World: Empirical Evidence from GitHub2014
- Programmers' build errors: a case study (at google)2014
- Modeling continuous integration practice differences in industry software developmentJournal of Systems and Software, 2014
- An Empirical Study of API Stability and Adoption in the Android Ecosystem2013
- AUC-RF: A New Strategy for Genomic Profiling with Random ForestHuman Heredity, 2011
- Scaling Continuous Integration2004
- Laws of software evolution revisited1996