Automated Penetration Testing with LLM Agents and Classical Planning

Lingzhi Wang; Xinyi Shi; Ziyu Li; Yi Jiang; Shiyu Tan; Yuhao Jiang; Junjie Cheng; Wenyuan Chen; Xiangmin Shen; Zhenyuan LI; Yan Chen

arXiv · 2512.11143

Automated Penetration Testing with LLM Agents and Classical Planning

Lingzhi Wang·Xinyi Shi·Ziyu Li·Yi Jiang·Shiyu Tan·Yuhao Jiang·Junjie Cheng·Wenyuan Chen·Xiangmin Shen·Zhenyuan LI·Yan Chen

View original source ↗Read source document ↗Export metadata ↓

Abstract

While penetration testing plays a vital role in cybersecurity, achieving fully automated, hands-off-the-keyboard execution remains a significant research challenge. In this paper, we introduce the "Planner-Executor-Perceptor (PEP)" design paradigm and use it to systematically review existing work and identify the key challenges in this area. We also evaluate existing penetration testing systems, with a particular focus on the use of Large Language Model (LLM) agents for this task. The results show that the out-of-the-box Claude Code and Sonnet 4.5 exhibit superior penetration capabilities observed to date, substantially outperforming all prior systems. However, a detailed analysis of their testing processes reveals specific strengths and limitations; notably, LLM agents struggle with maintaining coherent long-horizon plans, performing complex reasoning, and effectively utilizing specialized tools. These limitations significantly constrain its overall capability, efficiency, and stability. To address these limitations, we propose CHECKMATE, a framework that integrates enhanced classical planning with LLM agents, providing an external, structured "brain" that mitigates the inherent weaknesses of LLM agents. Our evaluation shows that CHECKMATE outperforms the state-of-the-art system (Claude Code) in penetration capability, improving benchmark success rates by over 20%. In addition, it delivers substantially greater stability, cutting both time and monetary costs by more than 50%.

Explore related subjects

cs.CR

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Lingzhi Wang, Xinyi Shi, Ziyu Li, Yi Jiang, Shiyu Tan, Yuhao Jiang, Junjie Cheng, Wenyuan Chen, Xiangmin Shen, Zhenyuan LI, Yan Chen. 2025-12-11. Automated Penetration Testing with LLM Agents and Classical Planning. https://arxiv.org/abs/2512.11143

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

WAInjectBench: Benchmarking Prompt Injection Detections for Web Agents

Multiple prompt injection attacks have been proposed against web agents. At the same time, various methods have been developed to detect general prompt injection attacks, but none have been systematically evaluated for web agents. In this work, we bridge this gap by presenting the first comprehensive benchmark study on detecting prompt injection attacks targeting web agents. We begin by introducing a fine-grained categorization of such attacks based on the threat model. We then construct datasets containing both malicious and benign samples: malicious text segments generated by different attacks, benign text segments from four categories, malicious images produced by attacks, and benign images from two categories. Next, we systematize both text-based and image-based detection methods. Finally, we evaluate their performance across multiple scenarios. Our key findings show that while some detectors can identify attacks that rely on explicit textual instructions or visible image perturbations with moderate to high accuracy, they largely fail against attacks that omit explicit instructions or employ imperceptible perturbations. Our datasets and code are released at: https://github.com/Norrrrrrr-lyn/WAInjectBench.

cs.CR↗

arXiv2026

MIRANDA: short signatures from a leakage-free full-domain-hash scheme

We present $\mathsf{Miranda}$, the first family of full-domain-hash signatures based on matrix codes. This signature scheme fulfils the paradigm of Gentry, Peikert and Vaikuntanathan ($\mathsf{GPV}$), which gives strong security guarantees. Our trapdoor is very simple and generic: if we propose it with matrix codes, it can actually be instantiated in many other ways since it only involves a subcode of a decodable code (or lattice) in a unique decoding regime of parameters. Though $\mathsf{Miranda}$ signing algorithm relies on a decoding task where there is exactly one solution, there are many possible signatures given a message to sign and we ensure that signatures are not leaking information on their underlying trapdoor by means of a very simple procedure involving the drawing of a small number of uniform bits. In particular $\mathsf{Miranda}$ does not use a rejection sampling procedure which makes its implementation a very simple task contrary to other $\mathsf{GPV}$-like signatures schemes such as $\mathsf{Falcon}$ or even $\mathsf{Wave}$. We instantiate $\mathsf{Miranda}$ with the famous family of Gabidulin codes represented as spaces of matrices and we study thoroughly its security (in the EUF-CMA security model). For~$128$ bits of classical security, the signature sizes are as low as~$90$ bytes and the public key sizes are in the order of~$2.6$ megabytes.

cs.CR↗

arXiv2026

The Role of Learning in Attacking ML-based Network Intrusion Detection

Machine Learning-based Network Intrusion Detection Systems (ML-NIDS) can be bypassed by rudimentary adversarial perturbations. Recent work has focused on identifying where such perturbations can realistically be applied by a host-side adversary. Yet every one of these attacks produces perturbations the same way: searching from scratch for every flow. The cost of an attack therefore grows in lockstep with the number of flows it must perturb, and real networks produce them by the tens of millions. In this paper, we show that using reinforcement learning to train lightweight perturbation-generating policies lets an adversary amortize that cost across flows it perturbs. Counting every detector query and every second an attack spends, training included, we compare learned policies against gradient, query-based, and random search across six ML-NIDS environments at two operating points, under both evasion and alert inflation. One successful adversarial example costs a learned policy 1.5 to 18 detector queries against 52 to 1,100 for the strongest search baseline, and the policy amortizes its training cost after 76 to 1,622 examples, a volume a monitored link produces in seconds of traffic. We further find that the RL formulation literature adopts by default is unnecessary for evasion, that the policy conditions on the flow it is given rather than converging on a fixed perturbation, and that it transfers to detectors and traffic it never trained against. The value of learning to attack ML-NIDS is therefore not a matter of effectiveness, but of scale.

cs.CR↗