Wake up and smell the coffee
Authors:
- Stephen M. BlackburnSBStephen M. Blackburn
- Robin GarnerRGRobin Garner
- Martin HirzelMHMartin Hirzel
- Antony HoskingAHAntony Hosking
Open Access
- 1 August 2008
- journal article
- Published by Association for Computing Machinery (ACM) in Communications of the ACM
- Vol. 51 (8) , 83-89
Abstract
Evaluation methodology underpins all innovation in experimental computer science. It requires relevant workloads, appropriate ex- perimental design, and rigorous analysis. Unfortunately, method- ology is not keeping pace with the changes in our field. The rise of managed languages such as Java, C#, and Ruby in the past decade and the imminent rise of commodity multicore architectures for the next decade pose new methodological challenges that are not yet widely understood. This paper explores the consequences of our collective inattention to methodology on innovation, makes recom- mendations for addressing this problem in one domain, and pro- vides guidelines for other domains. We describe benchmark suite design, experimental design, and analysis for evaluating Java ap- plications. For example, we introduce new criteria for measuring and selecting diverse applications for a benchmark suite. We show that the complexity and nondeterminism of the Java runtime system make experimental design a first-order consideration, and we rec- ommend mechanisms for addressing complexity and nondetermin- ism. Drawing on these results, we suggest how to adapt method- ology more broadly. To continue to deliver innovations, our field needs to significantly increase participation in and funding for de- veloping sound methodological foundations.Keywords
This publication has 11 references indexed in Scilit:
- Statistically rigorous java performance evaluation2007
- Hierarchical Means: Single Number Benchmarking with Workload Cluster Analysis2007
- Hardware atomicity for reliable software speculation2007
- The DaCapo benchmarks2006
- The exigency of benchmark and compiler drift2006
- MicroLib: A Case for the Quantitative Comparison of Micro-Architecture Mechanisms2005
- Myths and realities2004
- Auckland central business district power failurePower Engineering Journal, 1998
- A metrics suite for object oriented designIEEE Transactions on Software Engineering, 1994
- Principal Components Analysis1989