Search arXiv⌕ Search
Home/Archive/arXiv

arXiv · 2610.25308

From Software History to Executable Regression Tasks: A System for Evaluating Software Agents

Ruikai Shi·Zhengran Zeng·Rui Xie·Shikun Zhang·Wei Ye
View original source ↗Read source document ↗Export metadata ↓

Abstract

As software agents increasingly generate and modify code, evaluating their ability to review changes and discover defects becomes important. Such evaluation needs executable project environments for investigation and reliable defect references supported by reproducible behavior. Existing code review benchmarks provide limited support for this combination. We present a system that constructs executable regression instances from historical introducing--fixing commit pairs and developer tests. The system extracts candidate tests from fixing-side changes, then uses constrained agents to transplant them and configure version-specific environments while preserving production code and test expectations. Independent replay and evidence-based review verify whether tests that fail on the introducing version and pass on the fixing version support the proposed regression relationship. Accepted instances retain both versions' replay recipes, validated tests, and execution evidence. Repair-supported analysis and human review add root-cause locations, tested repairs, and causal explanations. These instances support bug detection and localization in code review, and enable bug repair and regression-test generation on the same defects. The system constructs 172 validated instances from 522 candidates across 11 Python projects. We evaluate three software agents with ten models and additional reasoning-effort settings on 83 manually selected, enriched instances across the three tasks. Precise root-line localization remains challenging, while baseline combinations achieve repair and regression-test generation success rates of up to 68.7% and 77.1%, respectively. Leading configurations differ by task, and higher task performance does not necessarily yield greater API-cost efficiency. Our approach provides reusable executable evidence for studying agents' defect discovery, repair, and testing capabilities.

Explore related subjects

cs.SE

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Ruikai Shi, Zhengran Zeng, Rui Xie, Shikun Zhang, Wei Ye. 2026-10-18. From Software History to Executable Regression Tasks: A System for Evaluating Software Agents. https://arxiv.org/abs/2610.25308

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

When Agent Evaluation Becomes Unreliable: An Empirical Study of Oracle Failures and Configuration-Dependent Measurement in an LLM Agent Benchmark

Agent evaluation harnesses are software: they run a test suite, drive a test input generator, and decide outcomes with a test oracle. We audit them for defects a score table cannot show. First, a systematic oracle-integrity auditing approach - five state-integrity invariants and three known-answer controls - is measured by injecting seven oracle defect classes across two task domains and a synthetic domain. Every class is detected, and on the 57 separable variant-target combinations it detects 54; the repaired oracle yields no false positive. On a widely used agent benchmark, it exposes a reward function that resets the database to build a ground-truth hash and never restores it, so a harness scoring afterwards reads the reference solution's state: recorded labels report 4,194 database passes over 4,320 episodes, replay gives 2,883, and success falls from 94.3% to 65.5%. The correction is not a uniform offset; it removes 24 to 34 points from each agent and changes which agent this matrix measures as strongest. Second, more weakly, localising the apparatus is a property of the measurement: a 3,240-episode ablation associates the three documents with lower collapse in the tested configuration, without ordering them, and a second task domain reverses the language effect. Third, the method did not transfer to a second, independently implemented benchmark; we name the contract required. A replication package recomputes every analysis offline with no model calls.

cs.SE↗
arXiv2026

VeriEvolve: Evaluator-First Evolution for Tool-Using Agents

A central obstacle to self-improving agents is the absence of a reliable fitness signal for realistic stateful tasks. We introduce \textsc{VeriEvolve}, an evaluator-first evolution framework that constructs this missing evaluation layer before optimizing the agent. From task instruction, VeriEvolve synthesizes instruction-grounded challenge states and an executable evaluator that checks them. During each round of workflow evolution, the evaluator is frozen and serves as the fitness signal. Between rounds, VeriEvolve revises the evaluator by converting the end states of strong and weak runs into a pair of counterfactual test cases. Across all 30 MCPMark Filesystem tasks, VeriEvolve constructs 30 evaluators with 141 challenge states, achieving 141/141 agreement on the generated suites and 113/122 agreement on held-out states produced by executor agents. Evaluator-guided evolution increases externally verified success from 6/30 to 18/30 for GLM-5.2 and from 27/30 to 30/30 for Claude; refinement removes all three false positives that the weak executor exposes. Cross-environment studies on PostgreSQL, Playwright, Notion, and GitHub further expose complementary limits from observability, state representation, and search/execution capacity. These results show that executable evaluators can be synthesized and used to guide search when task success is observable and an external verifier is available for calibration.

cs.SE↗
arXiv2026

Symmetry-Oriented Testing Framework for Variational Quantum Algorithms

Testing quantum software is challenging because quantum states cannot be directly inspected. Variational quantum algorithms (VQAs) are a particularly important target as they are among the most promising approaches for near-term noisy quantum hardware. Many testing methods rely on classically computed results, which conflicts with the regime where quantum advantage is expected. Although prior studies mitigate this oracle problem, the core difficulty remains that existing approaches do not verify whether the execution deviates from the intended problem or whether the obtained solution is physically plausible. We propose a symmetry-oriented testing framework for VQAs that addresses this gap by using operators commuting with the target Hamiltonian as test specifications. The framework provides two complementary diagnostics: a commutator-based health check that monitors whether each execution step during optimization remains consistent with the symmetry of the target Hamiltonian, and a variance-based acceptance check that assesses the converged solution by examining whether the state is an eigenstate of the symmetry operators. Both diagnostics derive from the problem structure and do not require the exact solution. We evaluate the framework on representative user- and system-side faults, across multiple target systems, and under injected noise. Results show that the two diagnostics detect faulty behavior, provide informative signals of solution quality, and remain useful under noisy execution.

cs.SE↗