Search arXiv⌕ Search
Home/Archive/arXiv

arXiv · 2610.20027

LintEval: Benchmarking LLMs on Linter-Guided File Refactoring

Tongwei Deng·Jinyu Zhu·Yue Cao·Chenxu Wang·Yanzhen Zou·Bing Xie
View original source ↗Read source document ↗Export metadata ↓

Abstract

Linters are widely used in code maintenance and real-time IDE editing to surface lint alerts related to performance, style, error-prone, and other patterns in source files. However, existing code benchmarks do not explicitly evaluate linter-guided file refactoring, where models must reduce lint alerts in source files with correct initial functionality while satisfying functionality preservation. We introduce LintEval, a multilingual benchmark for linter-guided file refactoring across C++, Python, and Java, comprising 13,061 programs with correct initial functionality and 171,662 lint alerts extracted using language-specific linters. LintEval jointly evaluates functionality preservation, lint alert reduction, new alert avoidance, and edit minimality. Experiments reveal a persistent trade-off: closed-source commercial models reach up to 90.4% Reduction@P but achieve only 67.1%-68.3% Pass@1, while several open-source small models show low or negative Reduction@P. A controlled fine-tuning study further shows that supervision format matters: refactoring-oriented SFT raises Qwen2.5-Coder-7B Reduction@P from 24.7% to 76.0%. LintEval enables controlled evaluation of models that improve source files under functionality preservation.

Explore related subjects

cs.SEcs.CL

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Tongwei Deng, Jinyu Zhu, Yue Cao, Chenxu Wang, Yanzhen Zou, Bing Xie. 2026-10-14. LintEval: Benchmarking LLMs on Linter-Guided File Refactoring. https://arxiv.org/abs/2610.20027

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Securing High-Concurrency Ticket Sales: A Framework Based on Microservice

The railway ticketing system is one of the most important public service infrastructure. In peak periods such as holidays, it is often faced with the challenge of high concurrency scenarios because of a large number of users accessing at the same time. The traditional aggregation architecture can not meet the peak user requirements because of its insufficient fault tolerance and low ability. Therefore, the system needs to use microservice architecture for development, and add multiple security methods to ensure that the system can have good stability and data consistency under high concurrency scenarios, and can respond quickly to user requests. This paper introduces the use of B/S architecture and Spring Cloud to design and develop a railway ticket purchase system that can maintain stability and reliability under high concurrency scenarios, and formulate multiple security design methods for the system. This system integrates a range of functions, such as real-time train inquiries, dynamic seat updates, online seat selection, and ticket purchasing, effectively addressing common problems associated with offline ticket purchasing, such as long queues and delayed information. It enables a complete online process from inquiry and booking to payment and refunds. Furthermore, the "add passenger" function allows users to purchase tickets for others, extending the convenience of online ticketing to people with limited internet access. The system design prioritizes security and stability, while also focusing on high performance, and achieves these goals through a carefully designed architecture and the integration of multiple middleware components. After the completion of the system development, the core interface of the system is tested, and then the results are analyzed. The test data proves that the system has good ability and stability under high concurrency.

cs.SE↗
arXiv2026

Terminal Agents Suffice for Enterprise Automation

There has been growing interest in building agents that can interact with digital platforms to execute meaningful enterprise tasks autonomously. Among the approaches explored are tool-augmented agents built on abstractions such as Model Context Protocol (MCP) and web agents that operate through graphical interfaces. Yet, it remains unclear whether such complex agentic systems are necessary given their cost and operational overhead. We argue that a coding agent equipped only with a terminal and a filesystem can solve many enterprise tasks more effectively by interacting directly with platform APIs. We evaluate this hypothesis across diverse real-world systems and show that these low-level terminal agents match or outperform more complex agent architectures at a fraction of the cost. Our findings suggest that simple, flexible programmatic interfaces combined with strong foundation models should be the backbone of enterprise automation.

cs.SE↗
arXiv2026

T2J-BENCH: Benchmarking Coding Agents on Execution-Grounded PyTorch-to-JAX Repository Migration

Migrating a machine-learning codebase across frameworks--for example from PyTorch to JAX--is a routine but costly engineering task that is fundamentally cross-paradigm: eager, stateful, object-oriented modules must be rewritten as pure, jittable functions over an explicit parameter pytree. We ask whether coding agents can perform such whole-repository migrations correctly, and argue that existing code benchmarks, which grade unit-test pass rates or surface-level similarity to a reference translation, cannot answer this: they neither require the migrated system to \emph{behave} the same nor prevent a JAX reimplementation from faking equivalence by secretly importing PyTorch itself. We introduce T2J-BENCH, a benchmark of 63 whole-repository PyTorch-to-JAX migration tasks spanning 18 model families across both training and serving, where torch-free checkpoint conversion is itself part of the deliverable. T2J-BENCH grades execution-grounded functional equivalence: each candidate is run inside a separate, torch-free, offline verifier and scored against references frozen from the PyTorch original, under a hard independence gate and a continuous, threshold-free reward. Across twenty off-the-shelf agent--model configurations, the strongest recovers only 0.676 of the reward against a 0.980 oracle, with substantial numeric and behavioral divergence despite high spec pass rates. Finally, we contribute TWINHARNESS, a reference-grounded agent harness that turns the agent's executable PyTorch source into legitimate self-verification feedback; applied to an opus-5 agent it raises the reward from 0.649 to 0.723--the highest of any configuration--using only task-visible resources.

cs.SE↗