Search arXiv⌕ Search
Home/Archive/arXiv

arXiv · 2610.14096

SWEeper-Bench: Can Agents Discover Bugs in Interactive Software?

Yang Yao·Haozhe Chen·Bingyi Kang·Karthik R Narasimhan·Zhuang Liu
View original source ↗Read source document ↗Export metadata ↓

Abstract

Coding agents increasingly interact with users by generating interactive software on the fly rather than replying in text, yet many bugs in such software reveal themselves only after multi-step interactions. Today, users must discover these bugs and wait for fixes, making the development loop slow and expensive. Existing benchmarks typically assume the bugs have already been identified, leaving agents' ability to find interaction-dependent failures largely unevaluated. We introduce SWEeper-Bench, 200 tasks built from real bug fixes in 200 widely used open-source web applications. Each task gives the agent only the codebase and an open-ended request such as "Find and fix issues in Focalboard's board sharing and membership." Because these bugs are hard to detect from code alone, agents must design interactive tests to uncover them. An agentic verifier then judges each repair by what a user would observe in the browser. The best of 15 frontier agents passes only 59.0% of tasks. Through stage-by-stage analysis of agent trajectories, we show that the biggest bottleneck is identifying the buggy behavior: agents often reach the relevant feature but never trigger the failure or fail to recognize it. SWEeper-Bench measures progress toward agents that close this loop themselves, delivering software that works without waiting for user feedback.

Explore related subjects

cs.SE

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Yang Yao, Haozhe Chen, Bingyi Kang, Karthik R Narasimhan, Zhuang Liu. 2026-10-08. SWEeper-Bench: Can Agents Discover Bugs in Interactive Software?. https://arxiv.org/abs/2610.14096

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

DynamiQ: Unlocking the Potential of Dynamic Task Allocation in Parallel Fuzzing

We present DynamiQ, a full-fledged and optimized successor to AFLTeam that supports dynamic and adaptive parallel fuzzing. Unlike most existing approaches that treat individual seeds as tasks, DynamiQ leverages structural information from the program's call graph to define tasks and continuously refines task allocation using runtime feedback. This design significantly reduces redundant exploration and enhances fuzzing efficiency at scale. Built on top of the state-of-the-art LibAFL framework, DynamiQ incorporates several practical optimizations in both task allocation and task-aware fuzzing. We evaluate DynamiQ on eighteen targets from Magma and FuzzBench with a total budget of over 40 CPU years. It achieves the highest mean branch coverage on fourteen targets, with average relative gains ranging from 8.37% to 19.53% over five baseline fuzzers. Across our experiments, DynamiQ identifies nine previously unknown bugs.

cs.SE↗
arXiv2026

Measuring the Permission Gate: A Stress-Test Evaluation of Claude Code's Auto Mode

Claude Code's auto mode is the first deployed permission system for AI coding agents, using a two-stage transcript classifier to gate dangerous tool calls. Anthropic reports a 0.4% false positive rate and 17% false negative rate on production traffic. We present the first independent evaluation of this system on deliberately ambiguous authorization scenarios, i.e., tasks where the user's intent is clear but the target scope, blast radius, or risk level is underspecified. Using AmPermBench, a 128-prompt benchmark spanning four DevOps task families and three controlled ambiguity axes, we evaluate 253 state-changing actions at the individual action level against oracle ground truth. Our findings characterize auto mode's scope-escalation coverage under this stress-test workload. The end-to-end false negative rate is 81.0% (95% CI: 73.8%-87.4%), substantially higher than the 17% reported on production traffic, reflecting a fundamentally different workload rather than a contradiction. Notably, 36.8% of all state-changing actions fall outside the classifier's scope via Tier 2 (in-project file edits), contributing to the elevated end-to-end FNR. Even restricting to the 160 actions the classifier actually evaluates (Tier 3), the FNR remains 70.3%, while the FPR rises to 31.9%. The Tier 2 coverage gap is most pronounced on artifact cleanup (92.9% FNR), where agents naturally fall back to editing state files when the expected CLI is unavailable. These results highlight a coverage boundary worth examining: auto mode assumes dangerous actions transit the shell, but agents routinely achieve equivalent effects through file edits that the classifier does not evaluate.

cs.SE↗
arXiv2026

A Multi-Dimensional, Per-Pass Empirical Study of the LLVM Optimization Pipeline

Quantifying the marginal impact of individual optimization passes underpins phase ordering, pass selection, optimization design, and analysis of pass/hardware interactions. In LLVM, the standard backend for C/C++, Rust, and ML stacks via MLIR, interactions among passes, measurement noise, and pipeline scale make this difficult. We present a systematic empirical study of the LLVM -O3 pipeline. We decompose it into 113 cumulative per-pass prefixes and measure execution time, compile time, binary size, hardware counters, and RAPL energy on 30 PolyBench/C kernels (84,750 measurements). Transitions whose binary is byte-identical to their predecessor give us a built-in null distribution, which we use to calibrate every significance claim. On these compute-bound affine kernels, the pipeline is non-monotone: 36% of the transitions that significantly change runtime are regressions, found in 28 of 30 kernels. Gains are back-loaded (the median kernel needs 84.8% of the pipeline for 80% of its speedup) and concentrated in a small core of passes, a result that holds under both absolute and scale-free impact measures. IR instruction count is an unreliable runtime predictor, and runtime-targeted passes are de facto energy-targeted (30-60% savings). Finally, we bound the speedup lost to non-monotone behavior from above by 46.35% and decompose it: 9.7 points are measurement noise, 15.3 points are transient dips repaired within three passes, and only 21.3 points persist to the end of the pipeline. These findings inform pass pruning, cost-model calibration, and autotuning.

cs.SE↗