A Survey on Reinforcement Learning-Driven Adversarial Sample Generation for PE Malware
Open Access
- 13 June 2025
- journal article
- review article
- Published by MDPI AG in Electronics
- Vol. 14 (12) , 2422
Abstract
Malware remains a central tool in cyberattacks, and systematic research into adversarial attack techniques targeting malware is crucial in advancing detection and defense systems that can evolve over time. Although numerous review articles already exist in this area, there is still a lack of comprehensive exploration into emerging artificial intelligence technologies such as reinforcement learning from the attacker’s perspective. To address this gap, we propose a foundational reinforcement learning (RL)-based framework for adversarial malware generation and develop a systematic evaluation methodology to dissect the internal mechanisms of generative models across multiple key dimensions, including action space design, state space representation, and reward function construction. Drawing from a comprehensive review and synthesis of the existing literature, we identify several core findings. (1) The scale of the action space directly affects the model training efficiency. Meanwhile, factors such as the action diversity, operation determinism, execution order, and modification ratio indirectly influence the quality of the generated adversarial samples. (2) Comprehensive and sensitive state feature representations can compensate for the information loss caused by binary feedback from real-world detection engines, thereby enhancing both the effectiveness and stability of attacks. (3) A multi-dimensional reward signal effectively mitigates the policy fragility associated with single-metric rewards, improving the agent’s adaptability in complex environments. (4) While the current RL frameworks applied to malware generation exhibit diverse architectures, they share a common core: the modeling of discrete action spaces and continuous state spaces. In addition, this work explores future research directions in the area of adversarial malware generation and outlines the open challenges and critical issues faced by defenders in responding to such threats. Our goal is to provide both a theoretical foundation and practical guidance for building more robust and adaptive security detection mechanisms.Keywords
Funding Information
- Major Scientific and Technological Research Project of Henan Province (241110210200)
This publication has 62 references indexed in Scilit:
- Enhancing reinforcement learning based adversarial malware generation to evade static detectionAlexandria Engineering Journal, 2024
- Reinforcement learning based adversarial malware example generation against black-box detectorsComputers & Security, 2022
- Enhancing the insertion of NOP instructions to obfuscate malware via deep reinforcement learningComputers & Security, 2022
- AntiViruses under the microscope: A hands-on perspectiveComputers & Security, 2022
- A survey and taxonomy of program analysis for IoT platformsAin Shams Engineering Journal, 2021
- An IRL-based malware adversarial generation method to evade anti-malware enginesComputers & Security, 2021
- Enhanced DNNs for malware classification with GAN-based adversarial trainingJournal of Computer Virology and Hacking Techniques, 2021
- Recurrent neural network for detecting malwareComputers & Security, 2020
- IMCFN: Image-based malware classification using fine-tuned convolutional neural network architectureComputer Networks, 2020
- Survey of machine learning techniques for malware analysisComputers & Security, 2019