Search arXiv⌕ Search
Home/Archive/arXiv

arXiv · 2602.14337

LongCLI-Bench: A Preliminary Benchmark and Study for Long-horizon Agentic Programming in Command-Line Interfaces

Yukang Feng·Jianwen Sun·Zelai Yang·Jiaxin Ai·Chuanhao Li·Zizhen Li·Fanrui Zhang·Kang He·Rui Ma·Jifan Lin·Jie Sun·Yang Xiao·Sizhuo Zhou·Wenxiao Wu·Yiming Liu·Pengfei Liu·Yu Qiao·Shenglin Zhang·Kaipeng Zhang
View original source ↗Read source document ↗Export metadata ↓

Abstract

Recent advances in AI-assisted programming have empowered agents to execute complex workflows via command-line interfaces, however, existing benchmarks are limited by short task horizons, data contamination from GitHub scraping, and a lack of fine-grained evaluation metrics, fail to rigorously evaluate the long-horizon planning and execution capabilities essential for realistic software engineering. To address these gaps, we introduce LongCLI-Bench, a comprehensive benchmark designed to evaluate agentic capabilities across long-horizon, realistic tasks. We curated 20 high-quality, long-horizon tasks from over 1,000 computer science assignments and real-world workflows, covering four engineering categories: from scratch, feature addition, bug fixing, and refactoring. We propose a dual-set testing protocol for LongCLI-Bench, which measures requirement fulfillment (fail-to-pass) and regression avoidance (pass-to-pass), and incorporates step-level scoring to pinpoint execution failures. Extensive experiments reveal that even state-of-the-art agents achieve pass rates below 20% in LongCLI-Bench. Step-level analysis further indicates that the majority of tasks stall at less than 30% completion, highlighting that critical failures often occur in the early stages. Although self-correction offers marginal gains, human-agent collaboration through plan injection and interactive guidance yields significantly higher improvements. These results highlight that future research must emphasize the development of synergistic human-agent workflows alongside advances in agents' planning and execution capabilities to overcome key challenges in long-horizon task performance.

Explore related subjects

cs.SEcs.MA

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Yukang Feng, Jianwen Sun, Zelai Yang, Jiaxin Ai, Chuanhao Li, Zizhen Li, Fanrui Zhang, Kang He, Rui Ma, Jifan Lin, Jie Sun, Yang Xiao, Sizhuo Zhou, Wenxiao Wu, Yiming Liu, Pengfei Liu, Yu Qiao, Shenglin Zhang, Kaipeng Zhang. 2026-02-26. LongCLI-Bench: A Preliminary Benchmark and Study for Long-horizon Agentic Programming in Command-Line Interfaces. https://arxiv.org/abs/2602.14337

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Search-based Trace Diagnostic for Cyber-Physical Systems

Cyber-physical systems (CPS) development requires verifying whether system behaviors violate their requirements. This analysis often considers system behaviors expressed by execution traces and requirements expressed by signal-based temporal properties. When an execution trace violates a requirement, engineers must solve the trace diagnostic problem---they need to understand the cause of the breach. Automated trace diagnostic techniques aim to support engineers in the trace diagnostic activity. This paper proposes search-based trace diagnostic (SBTD), a novel trace diagnostic technique for CPS requirements. Unlike existing techniques, SBTD relies on evolutionary search. SBTD starts from a set of candidate diagnoses, applies an evolutionary algorithm to generate new candidate diagnoses (via mutation, recombination, and selection), and uses a fitness function to determine the qualities of these solutions. Then, a diagnostic generator step is performed to explain the cause of the trace violation. We implemented Diagnosis, an SBTD tool for signal-based temporal logic requirements expressed using the Hybrid Logic of Signals (HLS). We evaluated Diagnosis by performing 34 experiments for 17 trace-requirement combinations for property violations. We assessed the effectiveness of SBTD in producing informative diagnoses and its efficiency. Diagnosis achieved expert-aligned diagnoses for 29/34 experiments and scaled to the full HLS benchmark, whereas state-of-the-art literature remained restricted to a subset due to performance and language limitations. SBTD treats trace-checking as a black box, which makes the checker replaceable. Substituting our HLS checker for an STL monitor, e.g., RTAMT, reproduces on two requirements the diagnoses at two to three orders of magnitude lower per-check cost.

cs.SE↗
arXiv2026

Toward Secure Code Generation: Bridging Correctness and Security via Task-Adaptive Vulnerability Modeling and Execution-Based Benchmarking

Large language models (LLMs) are increasingly used for program synthesis, yet they often generate code that is functionally plausible but insecure. Progress in secure code generation has been hindered by benchmarks that are small, non-executable, leak mitigation details, or rely on noisy analyzers and subjective judgments, making it difficult to measure whether security improves without sacrificing correctness. We address these gaps with CodeSecEval, an execution-based benchmark for secure code generation, comprising 255 Python tasks spanning 77 CWE categories. Each task provides paired insecure and secure implementations together with executable functional and vulnerability-targeted security tests, enabling precise and reproducible evaluation of secure code generation and insecure-code repair. Building on CodeSecEval, we propose SecAwareCoder, an agent-based framework that shifts code generation toward secure-by-construction synthesis. SecAwareCoder performs task-adaptive threat modeling to identify security-sensitive regions and derive task-grounded vulnerability hypotheses, uses these hypotheses to guide both constraint-aware code generation and security-aware test synthesis, and leverages execution feedback for targeted refinement. Experiments across multiple LLM backbones show that SecAwareCoder consistently improves Pass@1 and security robustness over prompting and analyzer-driven baselines, narrowing the security--correctness gap in LLM code generation.

cs.SE↗
arXiv2026

Enabling Communication via APIs for Mainframe Applications

Mainframe systems continue to support critical applications across industries such as banking, retail, and healthcare. Exposing their functionality through Application Programming Interfaces (APIs) enables reuse and development of new applications, but identifying and implementing APIs for legacy code remains challenging. It requires understanding complex programs, separating dependent components, introducing new artifacts, and preserving functionality and Service Level Agreements (SLAs) such as Turnaround Time (TAT). We propose a framework for APIfication of legacy mainframe applications. Candidate APIs are identified from artifacts such as transactions, screens, control-flow blocks, inter-microservice calls, business rules, and data accesses. Static analyses, including liveness and reaching definitions, are then used to traverse the code and automatically compute API signatures consisting of request and response fields. We evaluated the framework through a qualitative survey of nine mainframe developers with an average of 15 years of experience, using the public GENAPP application and two industrial mainframe applications. The results show that the framework identifies additional candidate APIs and reduces implementation effort for APIfication. The API-signature computation has been incorporated into IBM watsonx Code Assistant for Z Refactoring Assistant. We further validated the identified APIs by executing them on an IBM Z mainframe system, demonstrating the practical viability of the approach.

cs.SE↗