Smart Security Audit: Reinforcement Learning with a Deep Neural Network Approximator
No ratings
Presented at CyberScience2020 2020 by
A significant challenge in modern computer security is the growing skill gap as intruder capabilitiesincrease, making it necessary to begin automating elements of penetration testing so analysts can contend with thegrowing number of cyber threats. In this paper, we attempt to assist human analysts by automating a single hostpenetration attack. To do so, a smart agent performs different attack sequences to find vulnerabilities in a targetsystem. As it does so, it accumulates knowledge, learns new attack sequences and improves its own internalpenetration testing logic. As a result, this agent (AgentPen for simplicity) is able to successfully penetrate hosts it hasnever interacted with before. A computer security administrator using this tool would receive a comprehensive,automated sequence of actions leading to a security breach, highlighting potential vulnerabilities, and reducing theamount of menial tasks a typical penetration tester would need to execute. To achieve autonomy, we apply anunsupervised machine learning algorithm, Q-learning, with an approximator that incorporates a deep neural networkarchitecture. The security audit itself is modelled as a Markov Decision Process in order to test a number of decisionmaking strategies and compare their convergence to optimality. A series of experimental results is presented to showhow this approach can be effectively used to automate penetration testing using a scalable, i.e. not exhaustive, andadaptive approach.