Leveraging Deep Reinforcement Learning for Automating Penetration Testing in Reconnaissance and Exploitation Phase

6:38 24/11/2022

Penetration testing is one of the most common methods for assessing the security of a system, application, or network. Although there are different support tools with great efficiency in this field, penetration testing is done mostly manually and relies heavily on the experience of the ethical hackers who are doing it, known as pentesters. This paper presents an automated penetration testing approach that leverages deep reinforcement learning (RL) to automate the penetration testing process, including the reconnaissance and exploitation phases. More specifically, the RL agent is trained with the A3C model to gain experience choosing an exact payload to exploit available vulnerabilities. Ad-ditionally, our RL-based pentesting tool has three main functions: information gathering, vulnerability exploitation, and reporting. The performance of this approach is benchmarked against real-world vulnerabilities in our experimental environments. After training with environmental settings, the RL agent can assist pentesters in quickly identifying vulnerabilities in their own servers. The RL-based approach can mitigate the problems of labor costs and hunger data for automating penetration testing in the system by learning how to execute exploits on its own. The more pentesters who use this tool, the more accurate the pentesting results will be. With outstanding results, this method proves that it can accumulate learning results from previous environments to successfully exploit vulnerabilities for the next exploit in another environment on the first try.

Software-defined networking (SDN) is a potential approach for modern network architecture, which has received great attention recently. SDN-based networks also face security issues, and they can become targets of cyberattacks. Cyber threat hunting is one of the security solutions proposed for early attack detection in SDN. Developing machine learning-based IDS...
In large-scale networks like the Industrial Internet of Things (IIoT), it is more important to monitor and enforce the security policy within an appropriate time due to the continuous widespread of cyberattacks. This is a tough challenge in traditional network architecture; thus, each network element's network management is unsuitable for...