Qixuan Yang | Search arXiv

arXiv2026

CertClaim: Towards Certified Model-Claim Verification for Coding-Agent Services

Coding-agent services are priced by the model they claim, making the verification of such model claims crucial for their users. In this paper, we reveal that existing model auditing methods rely on a latent assumption that the deployment is static, i.e., the claimed model is always served in one reference configuration. This assumption does not hold for coding-agent services, whose providers may legitimately change the sampling preset, the reasoning mode or the system prompt without changing the model. As a result, existing methods either falsely accuse honest providers or miss cheap derivatives of the claimed model. To address this problem, we propose a certified model-claim verification method (dubbed CertClaim). Instead of testing the service against one reference deployment, we make the authorized deployments part of the claim and test whether some authorized deployment explains all the evidence of the service, by calibrating multiple evidence channels in every authorized deployment and fusing them within each deployment. We provide theoretical analyses of its false-accusation rate and detection condition. Extensive experiments on eight claims, two agent harnesses and three task suites verify the effectiveness of CertClaim, which is valid in every setting while separating substitutes far better than existing methods, as well as its resistance to potential adaptive attacks.

cs.CR↗

arXiv2026

One Analyst Is Not Ground Truth: Grading Agent-Built Financial Models Against Observed Professional Practice

Language model agents can now construct complete financial models, but it remains unclear whether they have learned the financial judgment that gives those models meaning. Existing benchmarks often anchor correctness to an expert-authored solution (e.g., numerical targets or detailed rubrics) which introduces a implicit assumption: \emph{one expert solution can serve as ground truth.} This is appropriate when finance provides a unique answer, but not when judgment is required. Evidence from professional practice challenges this assumption: When financial models built by different analysts for the same company are graded against one another, the median score is only 0.33 under standard tolerances, \emph{revealing that single-reference grading confounds professional disagreement with error.} We therefore introduce GAUGE, a benchmark that decomposes financial-model evaluation into deterministic checks, rubric judgments and numerical rules. Built from 1,001 professional valuation spanning 922 companies and all 25 GICS industry groups, GAUGE checks mechanical properties deterministically and evaluates judgment-bearing quantities against ranges in professional practice. We then validate GAUGE as a measurement instrument by testing expertise ordering, held-out professional values, and robustness to LM judgements. Our findings show that current LM agents are far better at constructing financial models than at deriving the company-specific assumptions that drive valuation, a gap that even persists after fine-tuning.

cs.LG↗