Continuous validation of load test suites
Authors:
Open Access
- 22 March 2014
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 259-270
Abstract
Ultra-Large-Scale (ULS) systems face continuously evolving field workloads in terms of activated/disabled feature sets, varying usage patterns and changing deployment configurations. These evolving workloads often have a large impact, on the performance of a ULS system. Hence, continuous load testing is critical to ensuring the error-free operation of such systems. A common challenge facing performance analysts is to validate if a load test closely resembles the current field workloads. Such validation may be performed by comparing execution logs from the load test and the field. However, the size and unstructured nature of execution logs makes such a comparison unfeasible without automated support. In this paper, we propose an automated approach to validate whether a load test resembles the field workload and, if not, determines how they differ by compare execution logs from a load test and the field. Performance analysts can then update their load test cases to eliminate such differences, hence creating more realistic load test cases. We perform three case studies on two large systems: one open-source system and one enterprise system. Our approach identifies differences between load tests and the field with a precision of >75% compared to only >16% for the state-of-the-practice.Keywords
This publication has 27 references indexed in Scilit:
- Influential Cases in Multilevel Modeling: A Methodological CommentAmerican Sociological Review, 2010
- Automated anomaly detection and performance modeling of enterprise applicationsACM Transactions on Computer Systems, 2009
- Overview on Techniques in Cluster Analysis2009
- MapReduceCommunications of the ACM, 2008
- Replaying development history to assess the effectiveness of change propagation toolsEmpirical Software Engineering, 2006
- Automated test case generation for the stress testing of multimedia systemsSoftware: Practice and Experience, 2002
- Experience with performance testing of software systems: issues, an approach, and case studyIEEE Transactions on Software Engineering, 2000
- The automatic generation of load test suites and the assessment of the resulting softwareIEEE Transactions on Software Engineering, 1995
- Silhouettes: A graphical aid to the interpretation and validation of cluster analysisJournal of Computational and Applied Mathematics, 1987
- Time, clocks, and the ordering of events in a distributed systemCommunications of the ACM, 1978