Search arXiv⌕ Search
Home/Archive/arXiv

arXiv · 2610.26589

Beyond Attack Success Rate: Cost-Aware Metric for Evaluating Iterative LLM Jailbreaks

Rohan Menon·Chih-Hong Cheng
View original source ↗Read source document ↗Export metadata ↓

Abstract

Attack success rate (ASR) evaluates jailbreaks against large language models without accounting for repeated calls to the victim model. Yet each additional call gives the deployed system another opportunity to detect the attack and intervene. We introduce $\mathrm{ASR}_{\mathrm{STEP}}$, which penalizes additional victim-model calls while retaining binary success judgments. Our evaluation considers GCG~\cite{zou2023universal}, PAIR~\cite{chao2025jailbreaking} with a step budget, and a single-step PAP~\cite{zeng2024johnny} baseline on JailBreakBench across three model families. GCG's scores decrease by 7.0--8.7 percentage points after discounting, while PAP's remain unchanged. While for PAIR~\cite{chao2025jailbreaking}, we see a wider discounting range between 3.4-12 percentage points, showing that the "step" quantity differs per iterative strategy. Reporting both metrics makes the discount due to repeated victim-model interaction explicit. Code is available \hyperlink{https://github.com/SuELS-Group/beyond_asr}{here}.

Explore related subjects

cs.CRcs.CL

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Rohan Menon, Chih-Hong Cheng. 2026-10-19. Beyond Attack Success Rate: Cost-Aware Metric for Evaluating Iterative LLM Jailbreaks. https://arxiv.org/abs/2610.26589

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Can You Trust the Vectors in Your Vector Database? Black-Hole Attack from Embedding Space Defects

Vector databases serve as the retrieval backbone of modern AI applications, yet their security remains largely unexplored. We propose the Black-Hole Attack, a poisoning attack that injects a small number of malicious vectors near the geometric center of the stored vectors. These injected vectors attract queries like a black hole and frequently appear in the top-k retrieval results for most queries. This attack is enabled by a phenomenon we term centrality-driven hubness: in high-dimensional embedding spaces, vectors near the centroid become nearest neighbors of a disproportionately large number of other vectors, while this centroid region is nearly empty in practice. The attack shows that vectors in a vector database cannot be blindly trusted: geometric defects in high-dimensional embeddings make retrieval inherently vulnerable. Based on this insight, we propose four attack paths tailored to different attacker capabilities. Our experiments show that up to 94.4% of queries are successfully attacked. Additionally, we study two directions of defense: hubness mitigation and detection-based filtering. Hubness mitigation either significantly reduces retrieval accuracy or provides only limited protection, while the detection-based defense is effective against some attack paths but fails against others. A robust and adaptive defense thus remains an open problem, and our findings indicate that vector databases require more careful treatment of security.

cs.CR↗
arXiv2026

Anywhere, Any-Stymie: Remote Activation of Trojan Malware on LiDAR with Modulated Signals

LiDAR sensors are widely deployed in autonomous systems for safety-critical 3D perception. We identify a previously unexplored attack surface in which dormant malware, embedded in the LiDAR sensing pipeline, remains inactive during normal operation and is triggered externally after deployment, without access to sensor hardware or networking at attack time. This malware was developed in a closed research test environment with vendor technical support, rather than by exploiting an inherent production supply-chain vulnerability. To trigger the malware, we repurpose the sensor's own native dual-return signal processing as a covert channel for a modulated optical signal delivered into the environment. Once triggered, the malware performs false data injection and suppression on static and mobile platforms. Unlike optical spoofing, where the transmitted signal is itself the attack, here it carries only a small key that the sensor decodes with existing signal processing. We establish feasibility with static triggering at 300~ft and drive-by runs reaching 35 mph, and verify that injected artifacts are taken up as detections by a fine-tuned 3D object detector. On a deployed tactical autonomous vehicle running an unmodified production autonomy stack, we demonstrate both fail-safe and fail-dangerous impact: emergency stop, mission abort, and collision. These results motivate stronger integrity guarantees across LiDAR development and deployment pipelines.

cs.CR↗
arXiv2026

VeriRouter: Towards Verifiable LLM Routing

Large language model (LLM) routers are increasingly deployed to optimize quality, latency, and cost through dynamic model selection and request rewriting. However, a compromised router can cause ineligible selection on the control plane, unendorsed rewriting or unfaithful relay on the data plane, undermining LLM routing integrity. We present VeriRouter, a verifiable LLM routing framework. Under an endorsed public context binding a private semantic index and rewrite banks, VeriRouter combines ZK proofs with a provider-issued serving certificate to establish integrity across both planes while keeping the router's private routing assets hidden from clients. For model selection, we design an efficient ZK proof for semantic search. For rewriting, local checks enforce message preservation and additions from committed banks. On real-world workloads, VeriRouter's model-selection and rewrite proofs average 21.92~s and 4.24~s, yielding 1,307$\times$ and 101$\times$ speedups over the fastest completing baselines. Relative to serving without verification, VeriRouter adds only 4.08 s under per-turn selection and 0.11 s under session-pinned selection.

cs.CR↗