Can LLMs Replace Humans During Code Chunking?

Christopher Glasz; Emily Escamilla; Eric O. Scott; Anand Patel; Jacob Zimmer; Colin Diggs; Michael Doyle; Scott Rosen; Nitin Naik; Justin F. Brunelle; Samruddhi Thaker; Parthav Poudel; Arun Sridharan; Amit Madan; Doug Wendt; William Macke; Thomas Schill

arXiv · 2506.19897

Can LLMs Replace Humans During Code Chunking?

Christopher Glasz·Emily Escamilla·Eric O. Scott·Anand Patel·Jacob Zimmer·Colin Diggs·Michael Doyle·Scott Rosen·Nitin Naik·Justin F. Brunelle·Samruddhi Thaker·Parthav Poudel·Arun Sridharan·Amit Madan·Doug Wendt·William Macke·Thomas Schill

View original source ↗Read source document ↗Export metadata ↓

Abstract

Large language models (LLMs) have become essential tools in computer science, especially for tasks involving code understanding and generation. However, existing work does not address many of the unique challenges presented by code written for government applications. In particular, government enterprise software is often written in legacy languages like MUMPS or assembly language code (ALC) and the overall token lengths of these systems exceed the context window size for current commercially available LLMs. Additionally, LLMs are primarily trained on modern software languages and have undergone limited testing with legacy languages, making their ability to understand legacy languages unknown and, hence, an area for empirical study. This paper examines the application of LLMs in the modernization of legacy government code written in ALC and MUMPS, addressing the challenges of input limitations. We investigate various code-chunking methods to optimize the generation of summary module comments for legacy code files, evaluating the impact of code-chunking methods on the quality of documentation produced by different LLMs, including GPT-4o, Claude 3 Sonnet, Mixtral, and Llama 3. Our results indicate that LLMs can select partition points closely aligned with human expert partitioning. We also find that chunking approaches have significant impact on downstream tasks such as documentation generation. LLM-created partitions produce comments that are up to 20% more factual and up to 10% more useful than when humans create partitions. Therefore, we conclude that LLMs can be used as suitable replacements for human partitioning of large codebases during LLM-aided modernization.

Explore related subjects

cs.SE cs.AI

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Christopher Glasz, Emily Escamilla, Eric O. Scott, Anand Patel, Jacob Zimmer, Colin Diggs, Michael Doyle, Scott Rosen, Nitin Naik, Justin F. Brunelle, Samruddhi Thaker, Parthav Poudel, Arun Sridharan, Amit Madan, Doug Wendt, William Macke, Thomas Schill. 2025-06-24. Can LLMs Replace Humans During Code Chunking?. https://arxiv.org/abs/2506.19897

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Enhancing the Non-Functional Quality Compliance of LLM-Generated Code through Quality-Aware Preference Learning

Large Language Models (LLMs) have been widely adopted in commercial code completion engines, significantly enhancing coding efficiency and productivity. However, even functionally correct LLM-generated code may exhibit non-functional quality issues that violate coding standards and best practices, such as poor style and limited maintainability. To address this, we propose a framework for quality-aware preference learning that guides LLMs toward generating criteria-compliant code. Our approach consists of three phases. First, we construct a dataset of paired criteria-violating and criteria-compliant samples, where each pair contains code exhibiting a specific non-functional quality issue and its repaired version that resolves the issue. Second, we design an adaptive token weighting mechanism to emphasize quality-sensitive code regions. Third, we introduce a hybrid optimization objective that combines ranking loss with language modeling loss and KL divergence to enable effective comparative optimization. Extensive experiments on DeepSeek-Coder and Qwen2.5-Coder show that our method substantially improves compliance with the targeted non-functional quality criteria while maintaining functional correctness, achieving a 75.7% relative increase in Quality Reciprocal Score (QRS) on MBPP-sanitized for Qwen2.5-Coder. Fine-tuning a 7B model requires less than three hours, indicating strong practical viability. Ablation studies and a user study further support the effectiveness of the proposed framework.

cs.SE↗

arXiv2026

ResTest: Targeted Coverage of Residual Not-Covered Code Using Large Language Models for Web GUI Testing

Automated web GUI testing (AWGT) approaches explore web applications through GUI actions to achieve code coverage. However, existing approaches, whether random-based, model-based, or reinforcement-learning-based, often struggle to generate continuous and semantically meaningful action sequences for testing complex functionalities, limiting their achievable code coverage. Recent LLM-based approaches partially alleviate this problem but still fall short due to limited capability in inferring testable functionalities and low success rates in executing tasks on complex web applications. In this paper, we propose ResTest, a complementary approach that uses coverage-report-guided LLMs to target residual not-covered code left by existing AWGT approaches. ResTest first runs an existing AWGT approach to broadly explore the application while constructing a state transition graph with summarized information. Once coverage plateaus, ResTest employs an LLM to infer not-covered functionalities based on the state transition graph and coverage report, and then utilizes a specialized LLM-driven GUI agent to execute these functionalities in a targeted manner. Our evaluation on ten open-source web applications shows that ResTest improves three categories of AWGT approaches by 17.52\% to 24.36\% in average code coverage. Ablation studies further confirm the sustained coverage improvement capability and the contribution of each component.

cs.SE↗

arXiv2026

Measuring and Exploiting Contextual Bias in LLM-Assisted Security Code Review

Automated Code Review (ACR) systems integrating Large Language Models (LLMs) are increasingly adopted in software development workflows, ranging from interactive assistants to autonomous agents in CI/CD pipelines. In this paper, we study how LLM-based vulnerability detection in ACR is affected by the framing effect: the tendency to let the presentation of information override its semantic content in forming judgments. We examine whether adversaries can exploit this through contextual-bias injection (crafting PR metadata to bias ACR security judgments) as a supply-chain attack vector against real-world ACR pipelines. To this end, we first conduct a large-scale exploratory study across 6 LLMs under five framing conditions, establishing the framing effect as a systematic and widespread phenomenon in LLM-based vulnerability detection. We then design a realistic and controlled experimental environment, evaluating 33 CVEs across 20 real-world projects and two popular ACR pipelines (Claude Code and CodeRabbit), to assess the susceptibility of real-world ACR pipelines to vulnerability re-introduction attacks. We employ two attack strategies: a template-based attack inspired by prior related work, and a novel LLM-assisted refinement attack. We find that template-based attacks are ineffective and may even backfire, as direct biasing attempts raise suspicions. Our refinement attack, on the other hand, is successful in 32/33 (97%) cases, exploiting a fundamental asymmetry: attackers can iteratively refine attacks against a local clone of the review pipeline, while defenders have only one chance to detect them. Overall, our findings highlight the dangers of over-relying on ACR and stress the importance of human oversight and contributor trust in the development process.

cs.SE↗