Search arXiv⌕ Search

arXiv subjects

Zaheed Ahmed

Publications and source records attributed to Zaheed Ahmed.

4 recordsLinked to original sources

TensorMutation: Learning Weight-Space Mutations of Deep Learning Models on Neural Graphs

Mutation testing for deep learning (DL) faces a cost-realism trade-off. The pre-training approach produces realistic but computationally expensive mutants, whereas existing post-training operators modify trained models efficiently but produce mutants that less closely resemble real faults. This study asks whether the trade-off is necessary: can the weight-space effect of a fault-informed pre-training mutation operator be predicted from a trained model, without retraining it? We answer this question with TensorMutation, which represents a trained model as a neural graph and uses a mutation-conditioned graph neural network (GNN) to predict parameter changes. The GNN is trained offline and reused across unseen models so that TensorMutation can generate executable mutants without retraining. We construct a corpus of 25,252 original-mutant pairs from 71 multilayer perceptron (MLP) classifiers using 11 fault-informed pre-training operators, including 25 newly curated subjects. TensorMutation partially reproduces the behavioral effects of pre-training mutations on nine held-out subjects, and direct weight-space supervision significantly improves weight-space fidelity. On 12 real-fault subjects, TensorMutation achieves the highest median behavioral similarity among the evaluated approaches. The improvement over the post-training tool DeepMutation++ is statistically significant (Holm-adjusted p=0.018), whereas the differences from the pre-training tool DeepCrime and the norm-matched random control are not significant. These results show that fault-informed pre-training mutations leave learnable structure in weight space that transfers to unseen models.

cs.SE↗

An Empirical Study of the Realism of Mutants in Deep Learning

The core assumption behind using mutants as substitutes for real faults is that they represent them faithfully. In traditional software testing, this representativeness has been empirically studied by examining the behavioral relationship between mutant detection and real fault detection. In deep learning (DL), mutants are generated by applying mutation operators to either pre-training or post-training artifacts. However, the realism of mutants in DL remains largely unverified. To the best of our knowledge, this study presents the first empirical comparison of pre-training and post-training mutation approaches in DL with respect to realism. We introduce a statistical framework that quantifies realism using two complementary metrics: coupling strength and behavioral similarity to real faults. We evaluate the framework on four publicly available bug datasets: CleanML, DeepFD, DeepLocalize, and defect4ML. Mutants are generated using state-of-the-art tools representing both mutation approaches. Results show that pre-training mutants exhibit significantly stronger coupling and higher behavioral similarity to real faults than post-training mutants, indicating greater realism. Operator-level analysis shows that this advantage is systematic across individual mutation operators and ranks the operators by realism within each approach. However, the substantial computational cost of pre-training mutation underscores the need for more effective post-training operators that match or exceed the realism of pre-training mutants.

cs.SE↗

Quality-Driven Selective Mutation for Deep Learning

Mutants support testing and debugging in two roles: (i) as test goals and (ii) as substitutes for real faults. Hard-to-kill mutants provide better guidance for test improvement, while realism is essential when mutants are used to simulate real bugs. Building on these roles, selective mutation for deep learning (DL) aims to reduce the cost of mutant generation and execution by choosing operator configurations that yield resistant and realistic mutants. However, the DL literature lacks a unified measure that captures both aspects. This study presents a probabilistic framework to quantify mutant quality along two complementary axes: resistance and realism. Resistance adapts the classical notion of hard-to-kill mutants to the DL setting using statistical killing probabilities, while realism is measured via the generalized Jaccard similarity between mutant and real-fault detectability patterns. The framework enables ranking and filtering of low-quality mutation-operator configurations without assuming a specific use case. We empirically evaluate the approach on four datasets of real DL faults. Three datasets (CleanML, DeepFD, and DeepLocalize) are used to estimate and select high-quality operator configurations, and the held-out defect4ML dataset is used for validation. Results show that quality-driven selection reduces the number of generated mutants by up to 55.6% while preserving typical levels of resistance and realism under baseline-aligned selection thresholds. These findings confirm that dual-objective selection can lower cost without compromising the usefulness of mutants for either role.

cs.SE↗

A new perspective on the competent programmer hypothesis through the reproduction of bugs with repeated mutations

The competent programmer hypothesis states that most programmers are competent enough to create correct or almost correct source code. Because this implies that bugs should usually manifest through small variations of the correct code, the competent programmer hypothesis is one of the fundamental assumptions of mutation testing. Unfortunately, it is still unclear if the competent programmer hypothesis holds and past research presents contradictory claims. Within this article, we provide a new perspective on the competent programmer hypothesis and its relation to mutation testing. We try to re-create real-world bugs through chains of mutations to understand if there is a direct link between mutation testing and bugs. The lengths of these paths help us to understand if the source code is really almost correct, or if large variations are required. Our results indicate that while the competent programmer hypothesis seems to be true, mutation testing is missing important operators to generate representative real-world bugs.

cs.SE↗