TrustedCode: Evaluating Large Language Models for Developing Trusted Code through Automated Program Verification

Muhammad Danial; Muhammad Taimoor Khan

Home/Archive/arXiv

arXiv · 2610.30432

TrustedCode: Evaluating Large Language Models for Developing Trusted Code through Automated Program Verification

Muhammad Danial·Muhammad Taimoor Khan

View original source ↗Read source document ↗Export metadata ↓

Abstract

Recent reports, such as "Who is using AI to code?", indicate that approximately 30\% of Python code contributions to GitHub are AI-generated. Given the inherent unreliability of this code, formal verification emerges as a promising approach to establish trust in software correctness. Although leveraging large language models (LLMs) for formal verification has garnered increasing attention, practical adoption remains limited due to the challenges in writing formal specifications and verifying program correctness against them. Existing tools predominantly support verification for languages such as Java and C, with minimal support for Python. We present TrustedCode, an automated framework for verifying AI-generated Python code. TrustedCode uses LLMs to (i) type-check Python code, (ii) translate it into WhyML via the Why3 framework, and (iii) verify it against formal specifications using Why3 back-end automated theorem provers. Our method integrates chain-of-thought prompting through LangGraph to identify and address LLM limitations in specification generation and proving the specified programs. To evaluate TrustedCode, we conducted experiments on ten Python programs of varying complexity. Our analysis reveals that 50\% of the observed errors stem from domain knowledge gaps, 30\% from syntactic inaccuracies, and the remaining 20\% from semantic and type-related issues. These findings underscore the importance of targeted improvements in LLM-guided formal verification workflows for Python. TrustedCode is an open source tool available at: \href{github.com/mtaimoorkhan/TrustedCode}{github.com/mtaimoorkhan/TrustedCode}. The video demonstration is available at: \href{youtu.be/Nr5I2g8ec-s}{youtu.be/Nr5I2g8ec-s}

Explore related subjects

cs.SE cs.LO cs.PL

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Muhammad Danial, Muhammad Taimoor Khan. 2026-10-21. TrustedCode: Evaluating Large Language Models for Developing Trusted Code through Automated Program Verification. https://arxiv.org/abs/2610.30432

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

DynamiQ: Unlocking the Potential of Dynamic Task Allocation in Parallel Fuzzing

We present DynamiQ, a full-fledged and optimized successor to AFLTeam that supports dynamic and adaptive parallel fuzzing. Unlike most existing approaches that treat individual seeds as tasks, DynamiQ leverages structural information from the program's call graph to define tasks and continuously refines task allocation using runtime feedback. This design significantly reduces redundant exploration and enhances fuzzing efficiency at scale. Built on top of the state-of-the-art LibAFL framework, DynamiQ incorporates several practical optimizations in both task allocation and task-aware fuzzing. We evaluate DynamiQ on eighteen targets from Magma and FuzzBench with a total budget of over 40 CPU years. It achieves the highest mean branch coverage on fourteen targets, with average relative gains ranging from 8.37% to 19.53% over five baseline fuzzers. Across our experiments, DynamiQ identifies nine previously unknown bugs.

cs.SE↗

arXiv2026

Measuring the Permission Gate: A Stress-Test Evaluation of Claude Code's Auto Mode

Claude Code's auto mode is the first deployed permission system for AI coding agents, using a two-stage transcript classifier to gate dangerous tool calls. Anthropic reports a 0.4% false positive rate and 17% false negative rate on production traffic. We present the first independent evaluation of this system on deliberately ambiguous authorization scenarios, i.e., tasks where the user's intent is clear but the target scope, blast radius, or risk level is underspecified. Using AmPermBench, a 128-prompt benchmark spanning four DevOps task families and three controlled ambiguity axes, we evaluate 253 state-changing actions at the individual action level against oracle ground truth. Our findings characterize auto mode's scope-escalation coverage under this stress-test workload. The end-to-end false negative rate is 81.0% (95% CI: 73.8%-87.4%), substantially higher than the 17% reported on production traffic, reflecting a fundamentally different workload rather than a contradiction. Notably, 36.8% of all state-changing actions fall outside the classifier's scope via Tier 2 (in-project file edits), contributing to the elevated end-to-end FNR. Even restricting to the 160 actions the classifier actually evaluates (Tier 3), the FNR remains 70.3%, while the FPR rises to 31.9%. The Tier 2 coverage gap is most pronounced on artifact cleanup (92.9% FNR), where agents naturally fall back to editing state files when the expected CLI is unavailable. These results highlight a coverage boundary worth examining: auto mode assumes dangerous actions transit the shell, but agents routinely achieve equivalent effects through file edits that the classifier does not evaluate.

cs.SE↗

arXiv2026

A Multi-Dimensional, Per-Pass Empirical Study of the LLVM Optimization Pipeline

Quantifying the marginal impact of individual optimization passes underpins phase ordering, pass selection, optimization design, and analysis of pass/hardware interactions. In LLVM, the standard backend for C/C++, Rust, and ML stacks via MLIR, interactions among passes, measurement noise, and pipeline scale make this difficult. We present a systematic empirical study of the LLVM -O3 pipeline. We decompose it into 113 cumulative per-pass prefixes and measure execution time, compile time, binary size, hardware counters, and RAPL energy on 30 PolyBench/C kernels (84,750 measurements). Transitions whose binary is byte-identical to their predecessor give us a built-in null distribution, which we use to calibrate every significance claim. On these compute-bound affine kernels, the pipeline is non-monotone: 36% of the transitions that significantly change runtime are regressions, found in 28 of 30 kernels. Gains are back-loaded (the median kernel needs 84.8% of the pipeline for 80% of its speedup) and concentrated in a small core of passes, a result that holds under both absolute and scale-free impact measures. IR instruction count is an unreliable runtime predictor, and runtime-targeted passes are de facto energy-targeted (30-60% savings). Finally, we bound the speedup lost to non-monotone behavior from above by 46.35% and decompose it: 9.7 points are measurement noise, 15.3 points are transient dips repaired within three passes, and only 21.3 points persist to the end of the pipeline. These findings inform pass pruning, cost-model calibration, and autotuning.

cs.SE↗