Prudent Practices for Designing Malware Experiments: Status Quo and Outlook
Authors:
- Christian RossowCRChristian Rossow
- Christian J. DietrichCDChristian J. Dietrich
- Chris GrierCGChris Grier
- Christian KreibichCKChristian Kreibich
- Vern PaxsonVPVern Paxson
- Norbert PohlmannNPNorbert Pohlmann
- Herbert BosHBHerbert Bos
- Maarten van SteenMSMaarten van Steen
Open Access
- 1 May 2012
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- No. 10816011,p. 65-79
Abstract
Malware researchers rely on the observation of malicious code in execution to collect datasets for a wide array of experiments, including generation of detection models, study of longitudinal behavior, and validation of prior research. For such research to reflect prudent science, the work needs to address a number of concerns relating to the correct and representative use of the datasets, presentation of methodology in a fashion sufficiently transparent to enable reproducibility, and due consideration of the need not to harm others. In this paper we study the methodological rigor and prudence in 36 academic publications from 2006-2011 that rely on malware execution. 40% of these papers appeared in the 6 highest-ranked academic security conferences. We find frequent shortcomings, including problematic assumptions regarding the use of execution-driven datasets (25% of the papers), absence of description of security precautions taken during experiments (71% of the articles), and oftentimes insufficient description of the experimental setup. Deficiencies occur in top-tier venues and elsewhere alike, highlighting a need for the community to improve its handling of malware datasets. In the hope of aiding authors, reviewers, and readers, we frame guidelines regarding transparency, realism, correctness, and safety for collecting and using malware datasets.Keywords
This publication has 22 references indexed in Scilit:
- BitShred2011
- Automatic analysis of malware behavior using machine learningJournal of Computer Security, 2011
- Online Botnet Detection Based on Incremental Discrete Fourier TransformJournal of Networks, 2010
- Analyzing and Exploiting Network Behaviors of Malware2010
- Dispatcher2009
- Traffic Aggregation for Malware Detection2009
- Using Failure Information Analysis to Detect Enterprise Zombies2009
- What are our standards for validation of measurement-based networking research?ACM SIGMETRICS Performance Evaluation Review, 2008
- Botnet Detection Based on Network Behavior2007
- MANET simulation studiesACM SIGMOBILE Mobile Computing and Communications Review, 2005