How to Better Fit Reinforcement Learning for Pentesting: A New Hierarchical Approach

How to Better Fit Reinforcement Learning for Pentesting: A New Hierarchical Approach

연구 분야: Strategies

학회: European Symposium on Research in Computer Security

In response to the rapidly evolving landscape of Information Technology (IT) and Operational Technology (OT) systems, automated vulnerability assessment is gaining prominence. While traditional scripted approaches are common, supervised and unsupervised Machine Learning (ML) methods are gaining traction in cybersecurity. However, AI models’ generalization is often hindered by the scarcity and quality of the data representing the ever-evolving cyber-threats. Meanwhile, Reinforcement Learning (RL) approaches that rely on agents learning from living environments rather than static datasets, have shown promising generalization performances, while overcoming data availability issues. Nevertheless, striking the right balance between performance and fidelity remains an ongoing challenge. In this paper, we design the basis of an attacking agent which aims to help cybersecurity analysts to assess their systems’ vulnerabilities. This agent is based on a new Hierarchical RL (HRL) architecture combined to a sequential high-level policy with feature selection to take advantage of the structure of the observation and the action spaces. Our experiments on the CybORG RL environment show that our attacking agent is able to better tackle the complexity of Penetration Testing (PT) scenarios and adapt well to scenarios, significantly improving other state-of-the-art models in terms of efficiency and scalability.