Wake up and smell the coffee

Authors:

Abstract
Evaluation methodology underpins all innovation in experimental computer science. It requires relevant workloads, appropriate ex- perimental design, and rigorous analysis. Unfortunately, method- ology is not keeping pace with the changes in our field. The rise of managed languages such as Java, C#, and Ruby in the past decade and the imminent rise of commodity multicore architectures for the next decade pose new methodological challenges that are not yet widely understood. This paper explores the consequences of our collective inattention to methodology on innovation, makes recom- mendations for addressing this problem in one domain, and pro- vides guidelines for other domains. We describe benchmark suite design, experimental design, and analysis for evaluating Java ap- plications. For example, we introduce new criteria for measuring and selecting diverse applications for a benchmark suite. We show that the complexity and nondeterminism of the Java runtime system make experimental design a first-order consideration, and we rec- ommend mechanisms for addressing complexity and nondetermin- ism. Drawing on these results, we suggest how to adapt method- ology more broadly. To continue to deliver innovations, our field needs to significantly increase participation in and funding for de- veloping sound methodological foundations.