Automated Performance Deviation Detection across Software Versions Releases
Open Access
- 1 July 2017
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 450-457
Abstract
Performance is an important aspect and critical requirement in multi-process software architecture systems such as Google Chrome. While interacting closely with members of the Google Chrome engineering team, we observed that they face a major challenge in detecting performance deviations between releases, because of their very high release frequency and therefore limited amount of data on each. This paper describes a deep analysis on the data distributions followed by a comparative approach using median based confidence interval for software evaluation. This technique is capable of detecting performance related deviations. It is substantially different from the standard confidence interval, in that it can be used in the presence of outliers and random external influences since the median is less influenced by them. We conducted a bottom-up analysis, using stack traces in a very large pool of releases. The results show that our approach can accurately localize performance deviations at a function-level granularity, using a very small number of trace samples, nearby 5 runs.Keywords
This publication has 4 references indexed in Scilit:
- Diagnosing Performance Variations by Comparing Multi-Level Execution TracesIEEE Transactions on Parallel and Distributed Systems, 2016
- Automated root cause isolation of performance regressions during software development2013
- Automated detection of performance regressions using statistical process control techniques2012
- Confidence intervals rather than P values: estimation rather than hypothesis testing.BMJ, 1986