Search arXiv⌕ Search
Home/Archive/arXiv

arXiv · 2609.34469

The Last Mile Is the File: OfficeEditBench for Preservation-Aware Office Editing

Zhiwen Wu·Chengxu Wu
View original source ↗Read source document ↗Export metadata ↓

Abstract

A small Office edit creates two obligations: propagate every required update and leave protected state untouched. Updating too little leaves dependencies inconsistent; updating too much changes content the user did not authorize. We introduce OfficeEditBench, a 170-task benchmark for change-scoped maintenance of spreadsheets, presentations, and documents. Task contracts specify required updates, protected state, native structures, and applicable interaction requirements. Across 510 archived task-system outcomes from WorkBuddy, Doubao, and Codex, we distinguish file delivery, target completion, and verifier-defined acceptance. Hard package-valid delivery ranges from 92% to 100%, yet no selected output satisfies the complete contract. Case analysis highlights why local correctness is insufficient: an updated value can lose its generating formula, a revised rule can fail to reach related conclusions, and a new deadline can omit a retained prerequisite. These mechanisms connect artifact-level checks to the continued maintainability of Office files. We analyze maintenance failures while distinguishing frozen automatic verdicts from human acceptability. OfficeEditBench provides a testbed for completing required changes while preserving the logic and scope of existing work.

Explore related subjects

cs.SEcs.CL

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Zhiwen Wu, Chengxu Wu. 2026-09-28. The Last Mile Is the File: OfficeEditBench for Preservation-Aware Office Editing. https://arxiv.org/abs/2609.34469

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Beyond Accuracy: Behavioral Dynamics of Agentic Multi-Hunk Repair

Automated program repair has traditionally focused on single-hunk defects, overlooking multi-hunk bugs that are prevalent in real-world systems. Repairing these bugs requires coordinated edits across multiple, disjoint code regions, posing substantially greater challenges. We present the first systematic study of LLM-driven coding agents (Claude Code, Codex, Gemini-cli, and Qwen Code) on this task. We evaluate these four state-of-the-art agents on 404 multi-hunk bugs from the PolyHunk dataset, yielding 1,616 repair trajectories for large-scale behavioral analysis. We employ fine-grained metrics to assess localization, repair accuracy, regression behavior, and operational dynamics across agents. We find that localization capability varies substantially, with Codex achieving the highest success rate (75.3%) and Qwen Code the lowest (40.4%). Repair accuracy also differs widely, ranging from 26.98% (Qwen Code) to 92.82% (Claude Code), and consistently declines with increasing bug dispersion and complexity (hunk divergence and spatial proximity). High-performing agents (Claude Code and Codex) demonstrate superior semantic consistency, achieving positive average regression reduction, whereas lower-performing agents often introduce new test failures. Notably, agents do not fail fast; failed repairs consume substantially more resources (33%-440% more input tokens) and require longer execution time (35%-330%). Additionally, we developed Maple to provide agents with repository-level context. Empirical results show that Maple improves repair accuracy of Gemini-cli by ~21% through enhanced localization. By analyzing fine-grained metrics and trajectory-level analysis, this study moves beyond accuracy to explain how coding agents localize, reason, and act during multi-hunk repair. Our findings underscore the impact of bug divergence and spatial proximity on multi-hunk repair success for coding agents.

cs.SE↗
arXiv2026

Do Newer Models Produce Better Patches? A Non-Functional Quality Study

Repository-level coding benchmarks typically measure progress in model capability by comparing the resolved rates of later and earlier models. However, this focus overlooks whether the non-functional quality of their generated patches has also changed across model generations. This study investigates whether later models produce functionally correct patches with better non-functional characteristics than earlier models on comparable repository-level repair tasks. We conducted two case studies involving four Claude and DeepSeek models on SWE-bench Lite. Using the same SWE-agent functional repair setting, we evaluated the generated patches with CodeQL, CodeScene, CPU time, and peak memory. Our primary analysis compared the models on commonly resolved instances. The static analysis results showed that most CodeQL paired differences were zero and that no CodeQL or CodeScene comparison remained significant after Holm correction. CPU time differences were small and inconsistent across model families, while peak memory usage was slightly higher for the later models under the benchmark test workload, with small absolute differences. Differences in individual CodeQL rules and CodeScene categories varied across model families and did not survive multiple-comparison correction. Overall, later models resolved more instances but showed no consistent improvement in the measured non-functional indicators on tasks solved by both models. Through this study, we hope to encourage a more comprehensive evaluation of models' practical software engineering capabilities.

cs.SE↗
arXiv2026

Better Understanding, Better Fixes? A Study of Hallucination in LLM-based Automated Program Repair

Large language models (LLMs) have significantly advanced automated program repair (APR), yet existing evaluations remain largely result-centric and provide limited insight into hallucination during repair. In APR, hallucination may arise not only in final patches but also in the intermediate artifacts that guide patch generation. To address this gap, we perform a multi-layered analysis of hallucination throughout the APR process. Specifically, we characterize hallucination as the production of patches or intermediate artifacts that are not faithfully grounded in the available repair evidence. We examine repair hallucination in final patches and understanding hallucination in intermediate artifacts through three tasks, namely triggering testcase identification, line coverage prediction, and additional testcase generation. We then evaluate three representative LLMs on 832 Defects4J bugs through automatic evaluation and manual analysis. Our results show that both repair and understanding hallucinations remain prevalent. Across models and settings, only 21.0%-55.9% of generated patches pass the developer-written test suite. Moreover, although more accurate intermediate artifacts are generally associated with successful repairs, this relationship does not always hold. Manual analysis of 812 sampled repairs identifies repair hallucinations in 72.7% of cases, including patches that pass all available tests; incorrect causal localization and incorrect repair strategies account for 45.9% and 18.5% of these hallucinations, respectively. Meanwhile, models frequently misidentify triggering testcases, mispredict line coverage involving branching control flow, and generate additional testcases with missing bug-triggering conditions or incorrect expected behavior.

cs.SE↗