Take It From Here: Evaluating Coding Agents On Project Takeover

Dingxian Wang; Zhe Fu; Jiaqi Deng; Chao Yin; Ganesh Prabhu; Karthik Gomadam Rajagopal; Darwin Yi; Huan Huo; Guandong Xu; Xianzhi Wang; Andrew Rabinovich; Zhao Chen

arXiv · 2611.07307

Take It From Here: Evaluating Coding Agents On Project Takeover

Dingxian Wang·Zhe Fu·Jiaqi Deng·Chao Yin·Ganesh Prabhu·Karthik Gomadam Rajagopal·Darwin Yi·Huan Huo·Guandong Xu·Xianzhi Wang·Andrew Rabinovich·Zhao Chen

View original source ↗Read source document ↗Export metadata ↓

Abstract

Coding agents increasingly collaborate on long-term projects. When taking over a project, they need to recover currently applicable requirements from the interaction history and update them in response to new requests. However, most coding benchmarks do not include long interaction histories, and some evaluations that consider history still do not cover the full process from requirement reconstruction to project delivery. We therefore provide agents with the interaction history, current request, and pre-request project environment, ask them to make the requested changes, and assess takeover performance through delivery tests. Across six model-harness configurations and 37 targets eligible for executable validation, the mean complete-delivery rate under Full History is only 4.95%. We then use requirement atoms to independently diagnose a larger set of takeover targets, examining relevant information identification, pre-request state reconstruction, and current request processing. We further supply correct pre-request or target states to examine the remaining difficulties in requirement processing and project modification. The diagnostics show that none of the six configurations produces fully correct post-request requirement states, and a case illustrates omission of previously recovered constraints during updating. With correct target states supplied, the mean complete-delivery rate is 8.11%, with build and regression pass rates of 96.40% and 90.99%, but target-test success remains 9.01%. These results show difficulties in both forming target requirement states and implementing supplied targets in existing projects. Based on these findings, we introduce ReqMemBench, which combines requirement diagnostics with delivery validation to support systematic evaluation of project takeover.

Explore related subjects

cs.SE cs.MA

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Dingxian Wang, Zhe Fu, Jiaqi Deng, Chao Yin, Ganesh Prabhu, Karthik Gomadam Rajagopal, Darwin Yi, Huan Huo, Guandong Xu, Xianzhi Wang, Andrew Rabinovich, Zhao Chen. 2026-11-05. Take It From Here: Evaluating Coding Agents On Project Takeover. https://arxiv.org/abs/2611.07307

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Characterizing Robustness in Semantic Code Clone Detection

Semantic code clone detection seeks to recognize shared functionality across different implementations, making robustness to behavior-preserving changes central to its effectiveness. Prior studies have exposed failures under such changes, motivating a systematic examination of how sensitivity varies with the rewrites used for evaluation. We present Systematic Code Transformation (SCT), a framework organizing 51 operators by code level, target, and edit category, and evaluate 14 token, tree, graph, and prompt-based detectors on BigCloneBench, Google Code Jam, and OJClone. By comparing transformation groups, repeated application, and alternative constructions, we connect overall performance changes with specific rewriting conditions. Of 25 token, tree, and graph detector and dataset combinations starting above 90% F1, 21 lose at least 10 percentage points under the composition of all operators. These losses vary with transformation type and application intensity, but are not determined by edit counts or code growth alone. Even at matched locations and edit counts, alternative constructions can expose different errors despite similar aggregate losses. Further analysis distinguishes reduced score separability from changed fixed-threshold decisions with retained ranking ability, revealing different scope for threshold recovery. Overall, these findings show why benchmark effectiveness needs to extend to stable discrimination across equivalent implementations and how systematic transformation testing can assess that capability.

cs.SE↗

arXiv2026

Vibe Coding in Software Development: A Multivocal Literature Review

Vibe coding is an AI-assisted software development approach in which developers express intent in natural language and large language models generate code. To our knowledge, no review has synthesized academic and practitioner sources under a common protocol. We conducted a multivocal literature review of publications from January 2022 to 9 September 2026, identified through searches in October 2025 and September 2026. After assessing source quality and credibility, we analyzed 266 sources (76 peer-reviewed, 190 grey) across eight research questions: 146 were direct, 93 adjacent, and 27 background. Presence and outcome findings receive reporting-support and evidence-strength labels, respectively. The second author conducted twelve checks; pre-reconciliation agreement was low in several, but substantial for the final directness rule and moderate for the outcome fields, which were checked in full. Sources describe a generation-evaluation-revision loop, with developer work shifting from writing code toward directing, supervising, and validating. Productivity is reported in 109 of the 159 sources contributing to the outcome question, of which 22 provide assertions only; these counts indicate reporting, not effect size. Most comparable sources report short-term productivity gains and long-term maintainability costs, but both receive Weak evidence-strength labels because some comparable sources report opposing directions. Short-term quality and defects are labeled Weak. Long-term productivity, long-term quality, and short-term reproducibility are labeled Moderate. Prototyping is reported more often than production use, and user-interface work more often than data and machine-learning work, but task-domain evidence is sparse. Future work should evaluate safeguards, session-level dynamics, long-term maintainability, and production use.

cs.SE↗

arXiv2026

How Should Agents See Code? Rethinking Code Representations for Agentic Software Engineering

LLM agents are increasingly used for repository-level software engineering tasks. However, most agents still access repositories through raw files, lexical search, and shell commands, treating code primarily as text. This paper investigates map-based code-representation workflows as a design variable in agentic software engineering, examining how different repository views change agents' retrieval behavior, task success, token use, and latency. In this paper, we study five code-representation workflows: abstract syntax trees, call graphs, module dependencies, documentation-enriched search, and a hybrid workflow that combines them. Each workflow exposes a repository map that the agent can query during a task. We evaluate Claude Code with Haiku 4.5 against a raw-tool baseline on four repository-level tasks: fault localization with LinuxFLBench, repository-level code generation and repair with SWE-bench Pro, test generation with SWT-bench, and security patching with SecBench. The results show that representation workflows are most useful as token-saving tools when localization is the bottleneck, while their success gains narrow when the decisive step is patch or test authoring. On LinuxFLBench, all five workflows improve recall@10 over the baseline (0.757-0.829 vs. 0.717) and reduce token usage by 25-39%. On SWE-bench Pro, resolution remains close to the baseline (44.3-50.6% vs. 49.0%) while token usage drops by 17-26%. On SecBench, Callgraph has the highest patch-resolution rate in this sample (34.3% vs. 28.9%) and uses fewer tokens, but all workflows increase latency. On SWT-bench, Hybrid and Docs have the highest reproducing-test rates in this sample (47.8% and 47.2% vs. 44.7%), but all workflows use more tokens and take longer. Overall, map-based workflows change both what the agent inspects and how much it spends, but their benefits are task-dependent rather than universal.

cs.SE↗