Search arXiv⌕ Search
Home/Archive/arXiv

arXiv · 2610.02928

Discriminating Fixture Coverage in Agent-Infrastructure Verification Suites

Xin Xu·Siru Tao
View original source ↗Read source document ↗Export metadata ↓

Abstract

Invariant suites and runtime monitors increasingly gate agent deployment decisions, and the evidence offered for any particular suite is almost always a single observation: it passes an implementation believed correct and fails one believed broken. We measure what that observation is worth. Applying mutation analysis to an invariant suite for a multi-session agent state-projection layer, we first find that this standard validation certifies a suite in which a first-order mutant removing event-identity deduplication survives every check. We then freeze the repaired twelve-check suite, record its hash, and run it once against ten mutants specified by an adversarial reader who designed none of its fixtures: it kills five. Instrumenting the five survivors against the reference shows they fail in two distinct ways, not one. Three are never activated, because no fixture supplies an input on which the mutated code behaves differently at all. The other two corrupt internal state that no oracle in the suite can observe. The two modes need different repairs, and neither is visible from a pass/fail report. Treating the missing inputs as a coverage question, we enumerate seven discriminating dimensions of the input space, register in advance which are uncovered and which survivors they should explain, and add one fixture per uncovered dimension while reusing the existing oracles verbatim. All five survivors then die, each to the check written for its predicted dimension. We report this as a repair result on the same challenge set rather than a second held-out estimate, and give the artifact, including the frozen hash, the registered predictions, all mutants and the run logs, so the distinction is checkable.

Explore related subjects

cs.SEcs.AIcs.LG

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Xin Xu, Siru Tao. 2026-10-02. Discriminating Fixture Coverage in Agent-Infrastructure Verification Suites. https://arxiv.org/abs/2610.02928

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

CodePori: Large-Scale System for Autonomous Software Development Using Multi-Agent Technology

Existing Large Language Model (LLM)-based multi-agent systems are capable of executing tasks and providing data-driven recommendations, thereby enabling automation and decision support that assist practitioners in software development. However, existing studies have evaluated agents' performance on benchmark datasets, offering only binary pass-or-fail results, which provide limited insight into their practical applicability. There remains a lack of empirical research examining the potential and limitations of LLM-based agents in addressing challenging, real-world tasks, such as automated code generation for software systems. To this end, this study conducted a survey to empirically investigate the potential of LLM-based agents in software development, with participants evaluating the agents' performance in autonomous software development tasks. We employed a two-phase approach comprising (i) the development of a multi-agent system, CodePori, to automate code generation, and (ii) a survey-based evaluation to assess agent performance and explore its practical applicability to software development. Our results highlight that, while LLM-based multi-agent systems show potential for autonomous code generation, their successful integration requires addressing specific challenges (e.g., short-term memory limitations, hallucinations, and code smells) and incorporating a practitioner-centric perspective. The study also highlights the need to move beyond standard benchmarks for evaluating real-world applicability and identifies new opportunities for broader adoption in both industry and academia.

cs.SE↗
arXiv2026

SCLA: Automated Smart Contract Summarization via LLMs and Control Flow Prompt

Smart contract code summarization is crucial for efficient maintenance and vulnerability mitigation. While many studies use Large Language Models (LLMs) for summarization, their performance still falls short compared to fine-tuned models like CodeT5+ and CodeBERT. Some approaches combine LLMs with data flow analysis but fail to fully capture the hierarchy and control structures of the code, leading to information loss and degraded summarization quality. We propose SCLA, an LLM-based method that enhances summarization by integrating a Control Flow Graph (CFG) and semantic facts from the code's control flow into a semantically enriched prompt. SCLA uses a control flow extraction algorithm to derive control flows from semantic nodes in the Abstract Syntax Tree (AST) and constructs the corresponding CFG. Code semantic facts refer to both explicit and implicit information within the AST that is relevant to smart contracts. This method enables LLMs to better capture the structural and contextual dependencies of the code. We validate the effectiveness of SCLA through comprehensive experiments on a dataset of 40,000 real-world smart contracts. The experiment shows that SCLA significantly improves summarization quality, outperforming the SOTA baselines with improvements of 26.7%, 23.2%, 16.7%, and 14.7% in BLEU-4, METEOR, ROUGE-L, and BLEURT scores, respectively.

cs.SE↗
arXiv2026

Generalizing GUI Tests into Voice Assistant Capabilities through Behavioral Variation Discovery

Voice Assistants (VAs) and agentic assistants increasingly support app-specific tasks through natural-language interaction. However, these capabilities still require app-specific execution logic. Existing approaches either derive this logic dynamically from GUI context, require developers to define app functions, or reuse GUI tests that remain tied to a single concrete task instance. To address this limitation, we propose Test2VA, an approach that generalizes existing GUI tests into reusable app-specific VA capabilities. Test2VA adopts mutation as an exploration mechanism to discover plausible behavioral variations of test interactions, dynamically validates these variations on the running application with adapted assertions, and generalizes the validated task instances into capabilities that preserve the common task pattern while supporting broader user requests. We evaluate Test2VA on 100 Espresso GUI tests from ten real-world Android applications, covering 407 GUI events and 115 assertions. Across 396 reproduced events, Test2VA reduces 18,999 available mutation candidates to 1,272 predicted candidates before dynamic validation, identifies mutable events with 83.33% precision, 87.79% recall, and an 85.50% F1 score, and generates 65 VA capabilities. Among them, 64 execute successfully and 56 satisfy our capability quality requirements. We further analyze unsuccessful and low-quality cases to identify common causes of failure. Finally, we conduct a user study with 18 participants and 150 task observations to evaluate the generated capabilities in real app tasks. The VA successfully completes 123 observations, achieving an overall success rate of 82.0%, and 16 of 18 participants rate the VA as useful or very useful. These results show the feasibility of generalizing test-derived task logic into reusable app-specific VA capabilities.

cs.SE↗