SoK: Benchmarking Flaws in Systems Security
- 1 June 2019
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 310-325
Abstract
Properly benchmarking a system is a difficult and intricate task. Even a seemingly innocuous mistake can compromise the guarantees provided by a systems security defense and threaten reproducibility and comparability. Moreover, as many modern defenses trade security for performance, the damage caused by benchmarking mistakes is increasingly worrying. To analyze the magnitude of the phenomenon, we identify 22 benchmarking flaws that threaten the validity of systems security evaluations, and survey 50 defense papers published in top venues. We show that benchmarking flaws are widespread even in papers published at tier-1 venues; tier-1 papers contain an average of five benchmarking flaws and we find only a single paper in our sample without any benchmarking flaws. Moreover, the scale of the problem appears constant over time, suggesting that the community is not yet taking sufficient countermeasures. This threatens the scientific process, which relies on reproducibility and comparability to ensure that published research advances the state of the art. We hope to raise awareness and provide recommendations for improving benchmarking quality and safeguard the scientific process in our community.Keywords
This publication has 17 references indexed in Scilit:
- Repeatability in computer systems researchCommunications of the ACM, 2016
- High System-Code Security with Low Overhead2015
- Producing wrong data without doing anything obviously wrong!ACM SIGPLAN Notices, 2009
- What are our standards for validation of measurement-based networking research?ACM SIGMETRICS Performance Evaluation Review, 2008
- A nine year study of file system and storage benchmarkingACM Transactions on Storage, 2008
- SPEC CPU2006 benchmark descriptionsACM SIGARCH Computer Architecture News, 2006
- MANET simulation studiesACM SIGMOBILE Mobile Computing and Communications Review, 2005
- Challenges in computer architecture evaluationComputer, 2003
- The case for application-specific benchmarking2003
- How not to lie with statistics: the correct way to summarize benchmark resultsCommunications of the ACM, 1986