An Automated Penetration Testing Framework Based on Hierarchical Reinforcement Learning
Open Access
- 2 November 2024
- journal article
- research article
- Published by MDPI AG in Electronics
- Vol. 13 (21) , 4311
Abstract
Given the large action space and state space involved in penetration testing, reinforcement learning is widely applied to enhance testing efficiency. This paper proposes an automatic penetration testing scheme based on hierarchical reinforcement learning to reduce both action space and state space. The scheme includes a network intelligence responsible for specifying the penetration host and a host intelligence designated to perform penetration testing on the selected host. Specifically, within the network intelligence, an action-masking mechanism is adopted to shield unenabled actions, thereby reducing the explorable action space and improving the penetration testing efficiency. Additionally, the host intelligence employs an invalid discrimination mechanism, terminating testing after actions that do not alter system states, thereby preventing sudden increases in the number of neural network training steps for an action. An optimistic estimation mechanism is also introduced to select penetration strategies suited to various hosts, preventing training crashes due to value confusion between different hosts. Ablation experiments demonstrate that the host intelligence can learn different penetration strategies for varying penetration depths without significant fluctuations in training steps, and the network intelligence can coordinate with the host intelligence to perform network penetration steadily. This hierarchical reinforcement learning framework can detect network vulnerabilities more quickly and accurately, significantly reducing the cost of security policy updates.Keywords
Funding Information
- National Natural Science Foundation of China (62272129, 2023CXPT065, 2022TSGC2459)
- Key Research and Development Program of Shandong Province (Competitive Innovation Platform) (62272129, 2023CXPT065, 2022TSGC2459)
- Small and Medium-sized Enterprise Capacity Re-improvement Project of Shandong Province (62272129, 2023CXPT065, 2022TSGC2459)
This publication has 24 references indexed in Scilit:
- Modelling penetration testing with reinforcement learning using capture‐the‐flag challenges: Trade‐offs between model‐free learning and a priori knowledgeIET Information Security, 2023
- Hierarchical reinforcement learning for efficient and effective automated penetration testing of large networksJournal of Intelligent Information Systems, 2022
- A Closer Look at Invalid Action Masking in Policy Gradient AlgorithmsThe International FLAIRS Conference Proceedings, 2022
- Automating post-exploitation with deep reinforcement learningComputers & Security, 2021
- Survey of Attack Graph Analysis Methods from the Perspective of Data and Knowledge ProcessingSecurity and Communication Networks, 2019
- NIG-AP: a new method for automated penetration testingFrontiers of Information Technology & Electronic Engineering, 2019
- Recent Advances in Hierarchical Reinforcement LearningDiscrete Event Dynamic Systems, 2003
- Hierarchical Reinforcement Learning with the MAXQ Value Function DecompositionJournal of Artificial Intelligence Research, 2000
- Reinforcement Learning: An IntroductionIEEE Transactions on Neural Networks, 1998
- Transfer of learning by composing solutions of elemental sequential tasksMachine Learning, 1992