Analysis of redundancy and application balance in the SPEC CPU2006 benchmark suite
Authors:
Open Access
- 9 June 2007
- journal article
- Published by Association for Computing Machinery (ACM) in ACM SIGARCH Computer Architecture News
- Vol. 35 (2) , 412-423
Abstract
The recently released SPEC CPU2006 benchmark suite is expected to be used by computer designers and computer architecture researchers for pre-silicon early design analysis. Partial use of benchmark suites by researchers, due to simulation time constraints, compiler difficulties, or library or system call issues is likely to happen; but a random subset can lead to misleading results. This paper analyzes the SPEC CPU2006 benchmarks using performance counter based experimentation from several state of the art systems, and uses statistical techniques such as principal component analysis and clustering to draw inferences on the similarity of the benchmarks and the redundancy in the suite and arrive at meaningful subsets. The SPEC CPU2006 benchmark suite contains several programs from areas such as artificial intelligence and includes none from the electronic design automation (EDA) application area. Hence there is a concern on the application balance in the suite. An analysis from the perspective of fundamental program characteristics shows that the included programs offer characteristics broader than the EDA programs' space. A subset of 6 integer programs and 8 floating point programs can yield most of the information from the entire suite.Keywords
This publication has 15 references indexed in Scilit:
- Performance counters and development of SPEC CPU2006ACM SIGARCH Computer Architecture News, 2007
- Simulation of computer architectures: simulators, benchmarks, methodologies, and recommendationsIEEE Transactions on Computers, 2006
- Pin2005
- The use and abuse of SPEC: An ISCA panelIEEE Micro, 2003
- Designing computer architecture research workloadsComputer, 2003
- Automatically characterizing large scale program behavior2002
- SPEC CPU2000: measuring CPU performance in the New MillenniumComputer, 2000
- Evolution and evaluation of SPEC benchmarksACM SIGMETRICS Performance Evaluation Review, 1998
- SPEC as a performance evaluation measureComputer, 1995
- An overview of common benchmarksComputer, 1990