Unknown malcode detection via text categorization and the imbalance problem
Authors:
- 1 June 2008
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 156-161
Abstract
Todaypsilas signature-based anti-viruses are very accurate, but are limited in detecting new malicious code. Currently, dozens of new malicious codes are created every day, and this number is expected to increase in the coming years. Recently, classification algorithms were used successfully for the detection of unknown malicious code. These studies used a test collection with a limited size where the same malicious-benign-file ratio in both the training and test sets, which does not reflect real-life conditions. In this paper we present a methodology for the detection of unknown malicious code, based on text categorization concepts. We performed an extensive evaluation using a test collection that contains more than 30,000 malicious and benign files, in which we investigated the imbalance problem. In real-life scenarios, the malicious file content is expected to be low, about 10% of the total files. For practical purposes, it is unclear as to what the corresponding percentage in the training set should be. Our results indicate that greater than 95% accuracy can be achieved through the use of a training set that contains below 20% malicious file content.Keywords
This publication has 8 references indexed in Scilit:
- A Feature Selection and Evaluation Scheme for Computer Virus Detection2006
- Malware prevalence in the KaZaA file-sharing network2006
- Learning to detect malicious executables in the wild2004
- EditorialACM SIGKDD Explorations Newsletter, 2004
- Data mining methods for detection of new malicious executables2002
- On the Optimality of the Simple Bayesian Classifier under Zero-One LossMachine Learning, 1997
- Neural Networks for Pattern Recognition1995
- A vector space model for automatic indexingCommunications of the ACM, 1975