Search arXiv⌕ Search
Home/Archive/arXiv

arXiv · 2604.24525

Understanding the Limits of Automated Evaluation for Code Review Bots in Practice

Veli Karakaya·Utku Boran Torun·Baykal Mehmet Uçar·Eray Tüzün
View original source ↗Read source document ↗Export metadata ↓

Abstract

Automated code review (ACR) bots are increasingly used in industrial software development to assist developers during pull request (PR) review. As adoption grows, a key challenge is how to evaluate the usefulness of bot-generated comments reliably and at scale. In practice, such evaluation often relies on developer actions and annotations that are shaped by contextual and organizational factors, complicating their use as objective ground truth. We examine the feasibility and limitations of automating the evaluation of LLM-powered ACR bots in an industrial setting. We analyze an industrial dataset from Beko comprising 2,604 bot-generated PR comments, each labeled by software engineers as fixed/wontFix. Two automated evaluation approaches, G-Eval and an LLM-as-a-Judge pipeline, are applied using both binary decisions and a 0-4 Likert-scale formulation, enabling a controlled comparison against developer-provided labels. Across Gemini-2.5-pro, GPT-4.1-mini, and GPT-5.2, both evaluation strategies achieve only moderate alignment with human labels. Agreement ratios range from approximately 0.44 to 0.62, with noticeable variation across models and between binary and Likert-scale formulations, indicating sensitivity to both model choice and evaluation design. Our findings highlight practical limitations in fully automating the evaluation of ACR bot comments in industrial contexts. Developer actions such as resolving or ignoring comments reflect not only comment quality, but also contextual constraints, prioritization decisions, and workflow dynamics that are difficult to capture through static artifacts. Insights from a follow-up interview with a software engineering director further corroborate that developer labeling behavior is strongly influenced by workflow pressures and organizational constraints, reinforcing the challenges of treating such signals as objective ground truth.

Explore related subjects

cs.SEcs.AI

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Veli Karakaya, Utku Boran Torun, Baykal Mehmet Uçar, Eray Tüzün. 2026-04-27. Understanding the Limits of Automated Evaluation for Code Review Bots in Practice. https://arxiv.org/abs/2604.24525

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Specification Before Generation: A Pre-Registered, Five-Model Paired Evaluation of a Specification Frame for LLM-Generated Code in Money, Time, Idempotency, and Access Tasks

Code generated by large language models passes security checks at a rate that has barely moved in four years. In regulated backends, the defect classes that matter most are money arithmetic, time handling, retry safety, and access control. Teams answer with instruction files, yet the largest controlled study of instruction files to date found no benefit. This paper tests a narrower idea: generated code improves when the prompt carries a specification, a fixed preamble stating what must be true of the result. We pre-registered hypotheses, refuters, analysis code, and a one-shot generation rule, then ran 50 realistic backend tasks from finance, healthcare, and insurance practice through five frontier models from five vendor lineages, each task twice: bare, and preceded by a 267-word filled specification frame. Nine deterministic AST-based checkers scored the outputs. The Bandit security scanner, which knows nothing of the frame, scored them independently. The frame reduced defects in all five models (mean reduction 0.16 to 0.70 findings per task, every Holm-adjusted sign test significant, every bootstrap confidence interval excluding zero). Where the arms differed, the frame arm won 95 of 100 times. It never made any model worse in any domain. Bandit found 53 medium-or-high issues in the bare arm and 11 in the frame arm, in the same direction for every model. The effect was largest where a model's unprompted defaults were weakest: the frame supplies the discipline a model lacks. All 500 outputs, prompts, checkers, scoring code, and the pre-registration are published with a DOI, so any team can re-derive the result without trusting the author.

cs.SE↗
arXiv2026

Enhancing the Non-Functional Quality Compliance of LLM-Generated Code through Quality-Aware Preference Learning

Large Language Models (LLMs) have been widely adopted in commercial code completion engines, significantly enhancing coding efficiency and productivity. However, even functionally correct LLM-generated code may exhibit non-functional quality issues that violate coding standards and best practices, such as poor style and limited maintainability. To address this, we propose a framework for quality-aware preference learning that guides LLMs toward generating criteria-compliant code. Our approach consists of three phases. First, we construct a dataset of paired criteria-violating and criteria-compliant samples, where each pair contains code exhibiting a specific non-functional quality issue and its repaired version that resolves the issue. Second, we design an adaptive token weighting mechanism to emphasize quality-sensitive code regions. Third, we introduce a hybrid optimization objective that combines ranking loss with language modeling loss and KL divergence to enable effective comparative optimization. Extensive experiments on DeepSeek-Coder and Qwen2.5-Coder show that our method substantially improves compliance with the targeted non-functional quality criteria while maintaining functional correctness, achieving a 75.7% relative increase in Quality Reciprocal Score (QRS) on MBPP-sanitized for Qwen2.5-Coder. Fine-tuning a 7B model requires less than three hours, indicating strong practical viability. Ablation studies and a user study further support the effectiveness of the proposed framework.

cs.SE↗
arXiv2026

Chaos Engineering in the Wild: Findings from GitHub

Chaos Engineering (CE) enhances system resilience by deliberately injecting faults to uncover weaknesses before they cause outages. Despite the availability of many CE tools, little is known about how they are adopted and maintained in open-source software (OSS) projects. This study empirically characterizes the adoption, evolution, and practical use of CE tools in OSS projects, examining who adopts them, how they are maintained, and which fault types they target. We conducted a large-scale mining study of GitHub repositories associated with ten widely used CE tools. Starting from 5,410 candidate repositories, we systematically filtered and manually validated 1,275 records and analyzed repository metadata, commit histories, and documentation. We found that adoption is concentrated around a few tools, with Toxiproxy, Chaos Mesh, and Chaos Monkey accounting for 68.86% of the validated repositories. In terms of adopter context, development is the predominant repository purpose (56.55%), while industry represents the largest ownership category (34.82%), closely followed by personal repositories (33.49%). At the project level, activity varies substantially, with 51.42% of repositories having at most 50 commits. In terms of fault coverage, network faults (44.85%) and instance termination (29.96%) together account for 74.81% of the 2,410 observed fault instances, whereas application-level faults account for only 2.57%. Taken together, these findings suggest that practitioners should consider repository activity and fault coverage when selecting CE tools, while researchers should investigate the factors behind concentrated adoption and limited application-level experimentation.

cs.SE↗