Practical Automation of Penetration Testing with Agentic AI

No ratings

Presented at CODE BLUE 2025 by

Hiroaki Toyota

Existing LLM-based penetration testing research has achieved some success by designing multi-agent systems that combine autonomous thinking and behavior. However, most of this research has been limited to testing in virtual lab environments, and quantitative evaluation of their application to real-world environments and the automation of all phases from reconnaissance to report generation is insufficient. This research aims to achieve practical-level penetration testing automation, including reconnaissance, vulnerability analysis, exploitation, and report generation. Based on findings gained from last year’s collaborative research with the Takagi Laboratory at Meiji University (arXiv:2502.15506v1), we developed a multi-agent system using modern Agent AI. For the evaluation system, we collaborated with penetration testers with extensive field experience to create a practical environment. This presentation will present the latest trends in AI-based penetration testing automation and the specific benefits of its implementation. First, we will outline the research trends since 2023, when LLM was introduced, and in the main part, we will report on a performance evaluation of the constructed system compared with conventional tools and manual diagnosis. In addition to HackTheBox, the evaluation will use an environment simulating a real-world situation to quantitatively compare the degree of reproducibility, coverage, and efficiency. The presentation will also touch on the potential for automation using local LLM. Through this presentation, participants will gain a deeper understanding of the benefits and challenges of introducing AI agents and gain concrete guidelines for applying them to their own company’s security assessments.