Search arXiv⌕ Search
Home/Archive/arXiv

arXiv · 2505.19395

VADER: A Human-Evaluated Benchmark for Vulnerability Assessment, Detection, Explanation, and Remediation

Ethan TS. Liu·Austin Wang·Spencer Mateega·Carlos Georgescu·Danny Tang
View original source ↗Read source document ↗Export metadata ↓

Abstract

Ensuring that large language models (LLMs) can effectively assess, detect, explain, and remediate software vulnerabilities is critical for building robust and secure software systems. We introduce VADER, a human-evaluated benchmark designed explicitly to assess LLM performance across four key vulnerability-handling dimensions: assessment, detection, explanation, and remediation. VADER comprises 174 real-world software vulnerabilities, each carefully curated from GitHub repositories and annotated by security experts. For each vulnerability case, models are tasked with identifying the flaw, classifying it using Common Weakness Enumeration (CWE), explaining its underlying cause, proposing a patch, and formulating a test plan. Using a one-shot prompting strategy, we benchmark six state-of-the-art LLMs (Claude 3.7 Sonnet, Gemini 2.5 Pro, GPT-4.1, GPT-4.5, Grok 3 Beta, and o3) on VADER, and human security experts evaluated each response according to a rigorous scoring rubric emphasizing remediation (quality of the code fix, 50%), explanation (20%), and classification and test plan (30%) according to a standardized rubric. Our results show that current state-of-the-art LLMs achieve only moderate success on VADER - OpenAI's o3 attained 54.7% accuracy overall, with others in the 49-54% range, indicating ample room for improvement. Notably, remediation quality is strongly correlated (Pearson r > 0.97) with accurate classification and test plans, suggesting that models that effectively categorize vulnerabilities also tend to fix them well. VADER's comprehensive dataset, detailed evaluation rubrics, scoring tools, and visualized results with confidence intervals are publicly released, providing the community with an interpretable, reproducible benchmark to advance vulnerability-aware LLMs. All code and data are available at: https://github.com/AfterQuery/vader

Explore related subjects

cs.CRcs.AI

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Ethan TS. Liu, Austin Wang, Spencer Mateega, Carlos Georgescu, Danny Tang. 2025-05-26. VADER: A Human-Evaluated Benchmark for Vulnerability Assessment, Detection, Explanation, and Remediation. https://arxiv.org/abs/2505.19395

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

The shifted-prime Erdős-Wintner law for primitive-root determinant densities: extremal order, dimension zero, and Fourier decay

For a prime $p$, let $c(p)=\frac{φ(p-1)}{p-1}\prod_{j\ge1}(1-p^{-j})$, the limiting density of matrices over $\mathbb F_p$ with primitive-root determinant. Its limiting law over the primes is the classical continuous shifted-totient law on $[0,1/2]$. We prove Hausdorff dimension zero and vanishing lower and upper dyadic $L^q$ dimensions for $q>1$. Its image $μ_f$ under $x\mapsto-\log x$ is Rajchman. As $T\to\infty$, for $U_T$ uniform on $[0,T]$, $\log|\widehat{μ_f}(U_T)|/\log\log T\to-1$ in probability. For every $A>0$, $|\widehat{μ_f}(τ)|\le(\log\log T)^4/\log T$ outside a subset of $[0,T]$ of relative measure $O_A((\log T)^{-A})$. As $h\downarrow0$, $\sup_aμ_f([a,a+h])=\mathfrak S_2e^{-γ}/\log(1/h)+O(\log^{-2}(1/h))$, where $\mathfrak S_2$ is the twin-prime singular series; maximizing left endpoints lie within $h$ of $\log3$ for small $h$. We prove $\min_{p\le x}c(p)\sim e^{-γ}/\log\log x$ and $\limsup_{p\to\infty}(c(p)\log\log p)^{-1}=e^γ$. The limiting law of $\log(φ(p+1)/φ(p-1))$ has support $\mathbb R$ and Hausdorff dimension zero. For the classical law of $σ(p-1)/(p-1)$ on $[3/2,\infty)$, we prove dimension zero, a sharp left-endpoint asymptotic, and a Rajchman logarithmic image. Its odd-prime component has an entire Mellin transform of order one. Partial-factorization bounds yield certified asymptotic searches for fully splitting negacyclic number-theoretic transform primes with prescribed reciprocal-density bounds at fixed power-of-two length. We determine the second distinct squared norm of $A_{n_1}\otimes\cdots\otimes A_{n_k}$ for $k,n_i\ge2$, yielding exact cyclotomic codifferent shell gaps and a uniform smoothing asymptotic at $ε=2^{-cφ(m)}$ for $c>2\log_2(1+\sqrt6)$. These results are unconditional. An explicit unproved exponent-pair hypothesis yields $|\widehat{μ_f}(τ)|=O(1/\log\log|τ|)$.

cs.CR↗
arXiv2026

Studying Detection Rule Generation as a Unified Task

Security systems use detection rules to identify suspicious activity. Existing studies often investigate rule generation for specific security systems, devoting substantial effort to developing dedicated methods and evaluation setups. Such customization contributes to fragmented research, limiting method reuse and result comparability across systems. We therefore study detection rule generation as a unified task across diverse natural language inputs and rule languages. To support method reuse, we propose UniRule, which abstracts diverse rules into shared natural language representations for retrieval. To enable consistent evaluation, we introduce a protocol that compares rules under shared criteria and aggregates the results into method scores. Experiments demonstrate the effectiveness of UniRule and the reliability of the evaluation protocol. They also show that method performance in one setting can be predicted from results in others, with average error close to that obtained using that setting's own data. These findings support studying detection rule generation as a unified task.

cs.CR↗
arXiv2026

Analyzing Defensive Misdirection Against Model-Guided Automated Attacks on Agentic AI Systems

Agentic AI systems increasingly rely on language-model components to interpret instructions, process external data, invoke tools, and coordinate with other agents. These capabilities make prompt-injection and jailbreak attacks more consequential, especially as attackers adopt model-guided automation to scale probing, prompt refinement, and response evaluation. This work analyzes the resulting attack-defense setting through a probabilistic model of a target system, its defense mechanism, and the attacker's automated judge. Our analysis shows that conventional detect-and-block defenses can allow attacker success rate (ASR) to approach one as the query budget grows, since predictable refusals provide useful feedback to automated search. We then examine detect-and-misdirect, where detected malicious interactions receive controlled, non-operational responses designed to induce false-positive errors in the attacker's judge. This strategy reduces the positive predictive value of attacker-selected candidates and yields a bounded asymptotic ASR. We evaluate a proof-of-concept realization of this strategy through Contextual Misdirection via Progressive Engagement (CMPE), a lightweight conversational misdirection method designed to replace predictable refusal text with safe but strategically misleading responses in automated jailbreak settings. On jailbreak benchmarks, CMPE reduces estimated ASR upper bounds by up to two orders of magnitude and nearly eliminates verified attack success in end-to-end experiments with PAIR, GPTFuzz, and AutoDAN-Turbo.

cs.CR↗