Search arXiv⌕ Search
Home/Archive/arXiv

arXiv · 2610.05399

Adaptive Code Revision Attacks on AI Pull Request Reviewers

Jingzhi Gong·Jie M. Zhang·Gunel Jahangirova·Meng Wang
View original source ↗Read source document ↗Export metadata ↓

Abstract

Pull-request review protects software before new code reaches users, helping prevent vulnerabilities that could expose users to attacks. AI agents increasingly perform these reviews and explain which problems need fixing. However, for an attacker submitting vulnerable code, this feedback also reveals what changes may secure approval. Existing PR attacks seek such approval through persuasive text and comments while keeping executable code fixed. This leaves unclear whether an attacker can use the feedback to repair the reported problem while preserving a vulnerability in the revised code. We therefore conduct an empirical study of this threat using AFCRA (Adaptive Feedback-guided Code Revision Attack). To distinguish successful attacks from genuine repairs, we construct AFCRA-Bench from 159 disclosed vulnerabilities, with executable exploits to verify vulnerabilities in code. Across five-round interactions with Sonnet 5 and GPT-5.5 reviewers, AFCRA reaches success rates 2.5x and 12.5x those of the strongest evaluated text- or comment-based attack. Case studies of these successes show how reviewers accept repairs of reported problems while overlooking surviving vulnerabilities. These findings establish feedback-guided code revision as a threat to automated PR review. To address this threat, we derive actionable implications for researchers, AI providers, PR reviewers, and PR authors on securing AI-assisted development.

Explore related subjects

cs.SE

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Jingzhi Gong, Jie M. Zhang, Gunel Jahangirova, Meng Wang. 2026-10-04. Adaptive Code Revision Attacks on AI Pull Request Reviewers. https://arxiv.org/abs/2610.05399

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Can we find bugs using LLM-generated oracles?

Unit testing is vital in software development. Typically, a unit test consists of a test prefix and a test oracle which captures the developer's intended behaviour. Traditional test generation tools (e.g. Randoop and Evosuite) often produce oracles that mirror the program's actual behavior rather than the expected one, limiting their ability to automatically detect bugs as users must manually verify if the generated assertions are correct. Recent approaches leverage Large Language Models (LLMs), trained on vast datasets, to generate developer-like code and test cases. Although successful in generating tests, the question of whether such LLM-generated oracles can automatically find bugs, i.e., expected software behavior, remains unanswered. We conduct a controlled experiment to answer this question, by studying LLMs on two tasks, namely, test oracle classification and generation, and assessing whether LLM oracles capture the actual or the expected behavior. The study includes test cases and oracles written by developers and automatically generated for 24 Java repositories. Our findings show that LLM-based test generation approaches mainly capture the actual program behavior making bug detection difficult. We also find that LLMs are better at generating oracles than classifying them. Notably, LLM-generated oracles have a higher fault detection potential than the Evosuite ones.

cs.SE↗
arXiv2026

SELU: A Software Engineering Language Understanding Benchmark

Large Language Models (LLMs) have demonstrated remarkable capabilities in code understanding and generation. However, their effectiveness on non-code Software Engineering (SE) tasks remains underexplored. We present 'Software Engineering Language Understanding' (SELU), the first comprehensive benchmark for evaluating LLMs on 22 SE textual artifacts NLU tasks, spanning from identifying whether a requirement is functional or non-functional to estimating the effort required to implement a development task. SELU covers classification, regression, Named Entity Recognition (NER), and Masked Language Modeling (MLM) tasks, with data drawn from diverse sources such as issue tracking systems and developer forums. We fine-tune 22 open-source LLMs, both generalist and domain-adapted; and prompt two proprietary alternatives using zero-shot a 3-shot prompting strategies. Performance is measured using metrics such as F1-macro, SMAPE, F1-micro, and accuracy, and compared via the Bayesian signed-rank test. Our results show that fine-tuned models across various sizes and architectures perform best, exhibiting high mean performance and low across-task variance. Furthermore, domain adaptation via code-focused pre-training does not yield significant improvements and might even be counterproductive for developer communication tasks.

cs.SE↗
arXiv2026

Mut4All: Fuzzing Compilers via LLM-Synthesized Mutators Learned from Bug Reports

Mutation-based fuzzing is effective for uncovering compiler bugs, but designing high-quality mutators for modern languages with complex constructs (e.g., templates, macros) remains challenging. Existing methods rely heavily on manual design or human-in-the-loop correction, limiting scalability and cross-language generalizability. We present Mut4All, a fully automated, language-agnostic framework that synthesizes mutators using Large Language Models (LLMs) and compiler-specific knowledge from bug reports. It consists of three agents: (1) a mutator invention agent that identifies mutation targets and generates mutator metadata using compiler-related insights; (2) a mutator implementation synthesis agent, fine-tuned to produce initial implementations; and (3) a mutator refinement agent that verifies and corrects the mutators via unit-test feedback. Mut4All processes 1400 bug reports (700 Rust, 700 C++), yielding 444 Rust and 561 C++ mutators at ~$0.08 each via GPT-4o. Our customized fuzzer, using these mutators, finds 62 bugs in Rust compilers (44 new, 32 fixed) and 38 bugs in C++ compilers (17 new, 3 fixed). Mut4All outperforms existing methods in both unique crash detection and coverage, ranking first on Rust and second on C++.

cs.SE↗