Design and evaluation of an LLM-augmented autonomous web application penetration testing agent
Date Issued
May 2026
Author(s)
Pilatsis, Simos
Advisor
Abstract
Web applications generated by large language models are increasingly common, yet their security characteristics remain poorly understood. This dissertation presents the design, implementation, and evaluation of an autonomous penetration testing agent that combines a hybrid whitebox and blackbox reconnaissance pipeline with an LLM reasoning core to identify vulnerabilities in web applications without access to backend source code. The agent uses Llama 3.3 70B, accessed via the Groq inference API, to select attack modules, generate context-specific payloads, and make adaptive decisions between attack executions through a five-directive interpreter loop. A library of thirteen attack modules covering the OWASP Top 10 vulnerability classes was developed and evaluated against five Docker-isolated targets, including two full web applications generated by Claude Sonnet 4.5 and Gemini 3.1 Pro. Across ten independent runs per target, the agent confirmed six vulnerabilities with full consistency, achieving a detection rate of 54.5% over selected attacks and a false positive rate of 9.1%. The results demonstrate that
LLM-augmented hybrid testing produces more targeted attack selection than rule-based approaches, and that both LLM-generated targets contained exploitable vulnerabilities despite being generated without deliberate weakness injection.
LLM-augmented hybrid testing produces more targeted attack selection than rule-based approaches, and that both LLM-generated targets contained exploitable vulnerabilities despite being generated without deliberate weakness injection.
Affiliation
File(s)![Thumbnail Image]()
Name
Simos Pilatsis.bsc.2026.abstract.pdf
Size
135.69 KB
Format
Adobe PDF
Checksum (MD5)
c413cc9a96f422a5f8de5b2b53e43cee