Decoding the Secrets of Machine Learning in Malware Classification: A Deep Dive into Datasets, Feature Extraction, and Model Performance
Open Access
- 15 November 2023
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 60-74
Abstract
Many studies have proposed machine-learning (ML) models for malware detection and classification, reporting an almost-perfect performance. However, they assemble ground-truth in different ways, use diverse static- and dynamic-analysis techniques for feature extraction, and even differ on what they consider a malware family. As a consequence, our community still lacks an understanding of malware classification results: whether they are tied to the nature and distribution of the collected dataset, to what extent the number of families and samples in the training dataset influence performance, and how well static and dynamic features complement each other. This work sheds light on those open questions by investigating the impact of datasets, features, and classifiers on ML-based malware detection and classification. For this, we collect the largest balanced malware dataset so far with 67k samples from 670 families (100 samples each), and train state-of-the-art models for malware detection and family classification using our dataset. Our results reveal that static features perform better than dynamic features, and that combining both only provides marginal improvement over static features. We discover no correlation between packing and classification accuracy, and that missing behaviors in dynamically-extracted features highly penalise their performance. We also demonstrate how a larger number of families to classify makes the classification harder, while a higher number of samples per family increases accuracy. Finally, we find that models trained on a uniform distribution of samples per family better generalize on unseen data.Keywords
All Related Versions
Funding Information
- Ministerio de Ciencia, Innovación y Universidades (TED2021-132464B-I00 - PID2022-142290OB-I00)
- Agence Nationale de la Recherche (ANR-22-PECY0007 - ANR-22- PECY-0008)
- European Research Council (771844)
This publication has 17 references indexed in Scilit:
- MALDC: a depth detection method for malware based on behavior chainsWorld Wide Web, 2019
- MAAR: Robust features to detect malicious activity based on API calls, their arguments and return valuesEngineering Applications of Artificial Intelligence, 2017
- Deep neural network based malware detection using two dimensional binary program features2015
- Large-scale malware classification using random projections and neural networks2013
- Opcode sequences as representation of executables for data-mining-based unknown malware detectionInformation Sciences, 2013
- Image visualization based malware detection2013
- Prudent Practices for Designing Malware Experiments: Status Quo and Outlook2012
- A comparative assessment of malware classification using binary texture analysis and dynamic analysis2011
- PE-Miner: Mining Structural Information to Detect Malicious Executables in Realtime2009
- Using Entropy Analysis to Find Encrypted and Packed MalwareIEEE Security & Privacy, 2007