Robustness-Aware Evaluation and Enhancement of Mutation-Based Fuzzing for Bug Discovery

Abstract

Mutation-based fuzzing is widely used to discover software vulnerabilities, but its randomness complicates rigorous evaluation and reliable bug detection. Prior work measures this variability empirically but lacks a theory with computable convergence and sample-complexity guarantees. We address both problems. First, we estimate robustness from independent campaigns by measuring variation in bug-trigger rates after accounting for compute. For $M$ campaigns of length $T$, finite-trial error decreases at the standard $M^{-1/2}$ rate, while finite-length error is bounded when temporal bug correlations decay. We then introduce splitting, a black-box wrapper that copies a fuzzer's queue state after a bug trigger and continues from that state in multiple branches, directing more effort toward the discovered region. For any realized split tree, branching cannot reduce the raw number of bug-triggering events relative to a single continuation path. Expected detection also improves when states receiving more branches tend to yield more bugs later. In a simplified model, splitting reduces variance per unit compute when $p<\sqrt{2}-1$, where $p$ is the fraction of time spent in the bug region. At matched compute, splitting finds more real bugs per CPU-hour than the baseline in 38 of 40 Magma ground-truth cells (median +52\%; 34 of 38 individually significant) and never finds fewer distinct bugs. It changes CVE-2019-19926 from undetected (0/20 trials) to reliably detected (20/20; Fisher $p<10^{-4}$), with six additional detection improvements, five involving CVEs. On FuzzBench, splitting finds more unique bugs in 53 of 70 pairs and reduces cross-campaign variation in 66 of 70, with a median reduction of about $10\times$. Every branch counts toward the compute budget. With about 0.14\% overhead, splitting provides a practical way to measure and improve fuzzers.