Search arXiv⌕ Search

arXiv · 2610.20616

Harnessing MPI mutations for AI error detection

Abstract

MPI errors are challenging to identify despite the significant number of expert verification tools. Dynamic tools (i.e., requiring profiling) are computationally expensive and accurate in error detection, whereas static analysis (i.e., operating at source code or compilation) is computationally cheap but less accurate. Interestingly, the recent success of AI and LLMs offers an alternative to increase static analysis accuracy while preserving its low overhead. Yet current methods remain difficult to benchmark, too general, and poorly adapted to the specific challenges of high-performance computing. In this paper, we investigate how AI-powered tools can efficiently and accurately detect errors in real-world MPI applications. We propose a novel MPI Mutated Dataset (MMD), constructed from MPI programs extracted from thousands of open-source GitHub projects. After sorting and filtering these files, we inject errors that realistically emulate developers' mistakes using synthetic code mutations. We leverage the dataset to train different AI models and assess their generalization capabilities against standard verification tools. We train compact multilayer perceptrons (MLP) with AST-T5 embeddings (i.e., transformer only encoder network) to detect diverse MPI errors (from the MPI-BugBench (MBB) classification) including CallOrdering or InvalidParameter, achieving detection accuracies up to 92.0%. In some cases, our method outperforms the expert prompted Qwen-3 Coder LLM and provides competitive results compared to established dynamic verification tools. We further evaluate Claude Sonnet 4.5 across different prompting strategies, showing that engineered prompts improve InvalidParam detection accuracy from 84.8% to 93.6%, surpassing the dynamic tool ITAC (87.0%). The full agentic pipeline achieves 73.9% overall multi-class accuracy and 87.6% bug detection recall on MBB. This paves the way for large-scale verification efforts that can use our dataset as a foundation to thoroughly investigate each MPI error in real world applications.

Explore related subjects

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Asia Auville, Tim Jammer, Eric Petit, Pablo de Oliveira Castro, Emmanuelle Saillard, Mihail Popov. 2026-07-22. Harnessing MPI mutations for AI error detection. https://arxiv.org/abs/2610.20616

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

Understanding Developer Information Needs for Software with Embedded Natural Language Prompts

Developers use foundation models (FMs) to create novel forms of intelligent software by writing natural language prompts embedded within a software's implementation---referred to as embedded prompts---that are also passed as a string within an FM's API. Developing software with embedded prompts requires developers to engage in iterative embedded prompt changes to obtain the desired behavior. While the information needs of traditional software have been investigated, the information needs of modern developers for embedded prompt changes are underexplored, despite these needs shaping how developers plan and justify embedded prompt changes. In this work, we perform a mixed-method exploratory study on the information needs of developers making changes to embedded prompts. We create, refine, validate, and apply a list of 50 information needs that developers have while making a change to an embedded prompt by interviewing 16 developers, observing 8 developers make prompt changes, surveying 50 developers, and evaluating 72 research and commercial tools. We find that embedded prompts are not well-supported by existing tools: 16 of the 50 information needs, including four of the most important ones, are unaddressed. We conclude with important design opportunities that should be addressed by future developer tools for embedded prompts.

cs.SE↗

Enhancing Neural Code Representation with Additional Context

Automated program comprehension underpins many software engineering tasks, from code summarisation to clone detection. While recent deep learning models have achieved strong results, they typically rely on source code alone, overlooking contextual signals such as version history or structural relationships. We conduct an empirical study of how enriching code representations with such signals affects model performance on four tasks: code clone detection, code classification, code summarisation, and vulnerability detection, using three datasets (SeSaMe, CodeSearchNet-Java, Vul4J). We evaluate five foundational code models (CodeBERT, GraphCodeBERT, CodeT5, PLBART, ASTNN) and seven code-specialised LLMs (CodeLlama, Qwen2.5-Coder, DeepSeek-Coder), comparing prompt-based context injection against fine-tuned use of context. Context improves classification for all five foundational models and, after fine-tuning, for most LLMs. For summarisation, the effect depends on the model family: version history improves all four foundational models on every metric, whereas prompt-level context helps two of the seven LLMs and hurts the other five. Combining context types reaches up to +21.48% macro-F1 in code classification and +15.04% F1 in clone detection. A human evaluation of CodeT5 summaries on 100 Java snippets ranks context-augmented summaries significantly higher for Accuracy and Content Adequacy. An exploratory vulnerability detection evaluation provides initial evidence of transfer to security-oriented tasks.

cs.SE↗

Cloud-OpsBench: A Reproducible Benchmark for Agentic Root Cause Analysis in Cloud Systems

LLM agents are explored for automating root cause analysis (RCA) in cloud-native systems, creating a need to evaluate both diagnostic correctness and the quality of the supporting investigation. Existing static benchmarks offer repeatable inputs but limited system-facing interaction, while live testbeds expose tools but hinder controlled comparison because incident evidence varies across runs; both paradigms focus primarily on final answers. To address these limitations, we present Cloud-OpsBench, an evaluation infrastructure for interactive and evidence-grounded cloud RCA. It comprises 754 runtime-verified cases across 57 fault types on two microservice workloads spanning application services and Kubernetes platform layers. Each fault is captured as a state snapshot and replayed through standard diagnostic interfaces, with outcome labels and diagnostic evidence graphs that enable matched comparisons and process-level analysis. Evaluating eleven LLMs on Cloud-OpsBench reveals that strong outcome accuracy often masks incomplete investigations: while Joint RCA Accuracy (JRA) reaches up to 0.78 on OnlineBoutique and 0.68 on TrainTicket, models identify the correct root cause without achieving evidence closure (e.g., in 33.1% and 29.4% of evaluated cases for GPT-5). These results demonstrate that final-answer accuracy alone cannot establish whether an agent has traced the failure and collected the evidence needed to confirm its root cause.

cs.SE↗