PSP-Mal: Evading Malware Detection via Prioritized Experience-based Reinforcement Learning with Shapley Prior
Open Access
- 4 December 2023
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 580-593
Abstract
With the widespread application of machine learning techniques in malware detection, researchers have proposed various adversarial attack methods to generate adversarial examples (AEs) of malware, thereby evading detection. Previous studies have shown that the reinforcement learning (RL) framework can enable black-box attacks by performing a sequence of function-preserving operations, which produces functional evasive malware samples. However, it is difficult to obtain the useful guidance and feedbacks from the environment for agent training in the black-box scenario, which results in the RL framework being unable to learn the effective evasion policy. In this paper, we propose the Shapley prior and establish a prior-guidance-based RL framework, namely PSP-Mal, to generate AEs against Portable Executable (PE) malware detectors. Our framework improves on existing methods in three aspects: 1) We explore feature effects of the black-box model by computing Shapley values and further propose the Shapley prior to represent the expected impact of operations. 2) A novel prioritized experience utilization mechanism is established regarding the Shapley prior guidance in the RL framework. 3) The actions are expanded into item-content pairs and we use the Thompson sampling to choose effective content, which helps to reduce randomness and ensure repeatability. We compare the attack performance of our framework with other methods, and experimental results demonstrate that our algorithm is more effective. The evasion rates of PSP-Mal against the LightGBM models trained on EMBER and SOREL-20M reach 76.88% and 72.03%, respectively.Keywords
Funding Information
- National Natural Science Foundation of China (62076251)
This publication has 31 references indexed in Scilit:
- Generic Black-Box End-to-End Attack Against State of the Art API Call Based Malware Classifiers2018
- Adversarial Malware Binaries: Evading Deep Learning for Malware Detection in Executables2018
- Audio Adversarial Examples: Targeted Attacks on Speech-to-Text2018
- Threat of Adversarial Attacks on Deep Learning in Computer Vision: A SurveyIEEE Access, 2018
- Deep neural network based malware detection using two dimensional binary program features2015
- Large-scale malware classification using random projections and neural networks2013
- A survey on heuristic malware detection techniques2013
- Malware Obfuscation Techniques: A Brief Survey2010
- Signature Generation and Detection of Malware Families2008
- Learning to detect malicious executables in the wild2004